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FOREWORD 


This  book  was  sponsored  by  the 
Operations  and  Maintenance  Office,  Office 
of  the  Chief  of  Engineers  (OCE)  as  part  of 
an  ongoing  nationwide  effort  to  realign  that 
organization  with  recent  Federal  Government 
initiatives  directed  toward  improving 
government  performance.  Much  to  the 
foresight  of  the  O&M  leadership,  these 
initiatives  were  actually  started  a  few  years 
(1988)  before  it  became  popular  to 
restructure  and  improve  government 
performance.  The  core  focus  of  the 
initiatives  was  directed  at  developing  an 
organizational  performance  change  and 
measuring  the  results. 

While  the  original  subject  of  this  text 
was  directed  at  developing  performance 
measures  exclusively  for  the  use  of  the  O&M 
organization  of  the  Army  Corps  of 
Engineers,  it  rapidly  evolved  into  an  all¬ 
purpose  tool  with  applicability  to  all 
government  elements.  Performance 
measurement  tools  are  universal  in  character. 
The  data,  context  and  units  of  measure  may 
vary  from  one  branch,  division,  or  office  to 
another,  but  the  problems  and  tools  of 
performance  measurement  are  alike. 

During  the  writing  of  the  text,  new 
topics  of  importance  surfaced.  The  original 
driving  force  behind  this  effort  spun  from  the 
Government  Performance  and  Results  Act 
(GPRA)  and  National  Performance  Review 
(NPR).  The  Army  Performance 


Improvement  Criteria  (APIC  or  the  so-called 
Baldrige  Criteria)  came  to  the  forefront 
adding  to  the  significance  of  performance 
measurement.  The  new  criteria  were  all 
added  to  the  text  without  loss  of  continuity  or 
intent.  In  fact,  all  three  factors  are 
complementary  and  have  been  integrated 
herein.  The  material  is  now  comprehensive 
and  designed  to  cover  all  needs  from  the 
O&M  community  within  the  Corps,  to  all 
government  organizations  and  offices,  and 
the  mandates  of  NPR,  GPRA  and  APIC. 
With  this  in  mind,  the  following  few  pages 
will  cover  the  historical  context  of  the 
development  of  the  O&M  effort  that  sparked 
and  funded  the  momentum  behind  this  text. 

The  following  paragraphs  were  taken 
from  the  initial  charter  for  the  program  and 
describe  the  history,  evolution,  and  plans  of 
the  process  to  date. 

Historical  Background 

The  steps  in  this  process  began  six  years 
ago  (1988)  when  performance  indicators 
were  first  introduced  to  the  Corps  of 
Engineers.  The  process  was  challenging  in 
retrospect,  but  established  the  first  set  of 
performance  indicators  that  were  used  in 
conjunction  with  Command  Management 
Reviews.  The  process  did  move  the  Corps 
community  forward  and  filled  a  management 
void  for  monitoring  performance. 
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In  1991,  the  Corps  of  Engineers  hired  a 
contractor  to  complete  a  comprehensive  study 
of  the  Operations  and  Maintenance  program. 
The  results  of  the  study  indicated  that  the 
Corps  was  moving  contrary  to  current 
management  theory.  The  development  of 
data  systems,  standards  of  information 
management,  workload  tracking,  and 
organizational  structure  reflected  the 
independent  tendencies  of  the  different 
Districts  and  Divisions.  The  results  were 
indicative  of  a  need  to  develop  Corps-wide 
standards  of  information  management  and 
performance  measurement  compatibility. 

Government  Actions  on  Performance 
Measurement 

Two  recent  visible  actions  were  taken  to 
improve  the  internal  management  of  the 
federal  government.  The  report  of  the 
National  Performance  Review  (Gore,  1993) 
recommends  scores  of  actions  to  be  taken  by 
specific  agencies  to  reduce  costs,  improve 
productivity,  and  provide  better  levels  of 
service  to  the  American  public.  This  call  for 
change  includes  establishing  performance 
goals  and  related  performance  measures  for 
the  conduct  of  all  “business”  within  federal 
agencies.  All  agencies  will  begin  developing 
and  using  measurable  objectives  and 
reporting  results  as  part  of  the  fiscal  year 
1996  (FY96)  budget  submissions. 

Performance  measurements  provide  the 
method  to  determine  if  past  goals  have  been 
met.  The  second  event  was  the  Government 
Performance  and  Results  Act  (PL  103-62), 
hereafter  referred  to  as  GPRA,  which 
initiates  internal  management  reforms  within 


the  federal  government  by  setting  program 
objectives  and  measuring  program 
performance  against  these  objectives.  Each 
federal  agency  will  submit  a  strategic  plan 
for  FY97  to  the  Director  of  the  Office  of 
Management  and  Budget  that  includes, 
among  other  specifications,  a  description  of 
how  performance  objectives  are  to  be 
achieved  and  measured. 

The  Government  Performance  and  Results 
Act 

The  purpose  of  the  GPRA  is  to  improve 
internal  management  of  federal  programs  and 
increase  public  confidence  by  systematically 
holding  agencies  accountable  for  results.  It 
further  aims  to  improve  congressional 
decision  making  by  providing  more  objective 
information  on  achieving  statutory  objectives 
and  the  relative  effectiveness  and  efficiency 
of  federal  programs  and  spending.  The 
GPRA  provides  for  a  series  of  pilot  projects 
to  test  the  costs  and  benefits  of  strategic 
planning,  performance  planning  and 
reporting,  and  performance  budgeting. 

The  Corps  of  Engineers  Operation  and 
Maintenance  community  has  been  selected  by 
the  Department  of  Defense  as  a  pilot  project 
to  set  up  procedures  and  implement 
performance  measurements  as  required  by 
the  GPRA.  To  meet  the  requirements  of  the 
pilot  program,  the  Corps  must  have  in  place 
a  strategic  plan,  a  performance  plan,  and  an 
evaluation  system  for  the  O&M  program. 
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The  Pilot  Project  Process 

The  Oregon  Department  of 
Transportation  (ODOT)  approach  to 
performance  management  is  recognized  as  a 
progressive  method  for  improving  the  Corps 
performance  measurement  process.  The 
O&M  Task  Force  has  recognized  the  value  of 
the  ODOT  approach  to  defining  Key  Result 
Areas  (KRA’s).  Adopting  this  process  will 
not  be  easy,  but  the  results  will  be  fruitful, 
and  the  timing  for  implementing  a  revised 
process  is  coincidental  with  the 
recommendations  of  the  National 
Performance  Review  and  the  GPRA. 

The  process  is  under  development  as  a 
pilot  program  for  Office  of  Management  and 
Budget  under  the  pilot  programs  provision  of 
the  GPRA.  The  process  will  set  standards 
for  other  Department  of  Defense  programs 
and  federal  agencies  on  how  to  develop  a 
coordinated,  multi-level  performance 
management  and  reporting  system.  The 
Corps  of  Engineers  Operation  and 
Maintenance  (O&M)  community  has 
embarked  on  the  development  of 
performance  measures  across  multiple 
functional  areas.  The  Performance  Measure 
Implementation  Plan  involves  a  four-tier 
approach: 

►  National  Leadership  Team 
Workshop 

►  National  Program  Proponent 
Workshops 

►  District  Leadership  Workshops 

►  Project  Site  Workshops 


The  four-tier  approach  was  developed  by 
the  Corps  Performance  Measurement  and 
Data  Management  Task  Force  in  conjunction 
with  ODOT  experts.  The  Operation  and 
Maintenance  Performance  Measurement 
Implementation  Plan,  dated  June  1994  (see 
Enclosure  1),  was  developed  by  the  Task 
Force  and  provides  a  detailed  description  of 
the  complete  implementation  program.  The 
O&M  community  has  adopted  a  project 
management  structure,  and  performance 
measurements  are  essential  to  support  that 
structure  and  other  proposed  improvements. 
The  four-tier  approach  promotes  the 
identification  of  specific  performance 
measures  at  each  management  level  of  the 
Corps  and  facilitates  review  and  feedback 
during  development. 

Tier  I  is  the  National  Leadership  Team 
and  is  central  to  the  development  of  the 
remaining  tiers,  (see  Figure  1).  Tier  II 
workshops  are  the  National  Program 
Proponent  workshops  and  develop  guidance 
for  the  Division  and  Headquarters  level  of 
management.  Additionally,  the  products  of 
the  Tier  II  workshops  will  provide  the  lower 
tiers  additional  guidance  for  the  development 
of  measures  at  the  District  and  project  levels. 
Tier  III  workshops  are  for  District 
Leadership  Workshops  and  Tier  IV 
workshops  are  for  the  project  site  level.  The 
four-tier  process  selected  by  the  O&M  Task 
Force  will  ensure  that  performance  measures 
reflect  the  management  stance  at  each  tier 
while  maintaining  alignment  with  the  Corps 
mission  and  vision  across  all  tiers.  The 
development  of  a  consensus  within  a  tier  is 
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crucial  to  the  performance  measurement 
process. 

The  implementation  of  this  process  is 
intended  to  create  a  cultural  change  within 
the  O&M  community.  Performance 
measures  will  not  be  developed  on  vague 
terms.  The  various  workshop  participants 
will  establish  pertinent  purpose  and  mission 
statements,  identify  component  programs  that 


support  their  specific  purpose,  develop 
relevant  KRA’s  and  formulate 
complementary  performance  measures.  This 
approach  requires  top  field  managers  to  help 
develop  KRA’s  and  “buy  in”  to  the  process. 
ODOT  is  providing  the  Corps  with  technical 
assistance  in  the  design  and  facilitation  of  a 
series  of  related  workshops  to  be  conducted 
during  the  summer  and  fall  of  1994. 
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PREFACE 


This  book  is  about  measuring 
organizational  performance.  It  is  the  only 
one  of  its  kind  in  that  it  serves  as  one 
comprehensive  resource  reference  written 
specifically  for  measuring  government 
performance  irrespective  of  the  mission.  It 
goes  well  beyond  the  use  of  numbers  and 
indexes  to  quantify  resource  inputs, 
efficiency,  productivity,  programs,  output 
and  outcomes.  What  struck  me  the  most 
when  writing  it  was  the  degree  of  overlap 
between  the  various  subjects  covered.  The 
material  is  very  holistic  in  character  in  that 
the  subjects  covered  emulate  a  tightly  knit 
web.  For  example,  it  was  not  possible  to 
write  about  improvement  in  “program 
performance”  by  focusing  solely  on 
program-related  activity  improvements.  An 
organization  is  a  system;  a  group  of 
processes  and  a  cluster  of  complex, 
integrated  and  interdependent  activities. 
Trying  to  improve  one  element  in  isolation 
without  working  on  the  entire  system  is  like 
rearranging  the  chairs  on  the  deck  of  the 
Titanic— the  end  results  will  be  the  same. 

While  the  intended  user  of  this  book  is 
the  U.S.  Army  Corps  of  Engineers,  the 
material  can  be  readily  adapted  to  any  local, 
state  or  federal  organization.  There  are 
about  twenty  concepts  which  keep 
reappearing  over  and  over  in  reading  the 
literature  on  TQM,  continuous  process 
improvements,  and  related  materials.  They 
represent  persistent  themes:  key  or  core 
concepts  needed  to  imprint  cultural  change  in 
organizational  behavior  and  are  found  in 


“world  class”  organizations.  They  include 
the  following: 

The  involvement  of  all  employees 

Measurement  or  metrics 

Integration 

Team  building 

Waste/Value  added 

Quality  defined  from  the  customer’s 
perspective 

Plan 

Simplify 

Vision 

Systematic  implementation 
Continuous  improvement 
Benchmarking 
Systems  and  Processes 
Cycle  time 
Empowerment 
Training /Education 
Prevention 
Cause  and  effect 
Results  /outcome 
Quality  as  strategy 


The  material  starts  out  with  a  brief 
description  of  organizational  and  planning 
characteristics  such  as  structure,  the  nature  of 
planning,  the  need  to  establish  a  vision,  a 
mission  statement,  program  targets  to  set 
goals  and  objectives  and  the  purposes  and 
uses  of  strategy  and  tactical  and  operational 
tools.  It  then  moves  into  its  prime  purpose— 
the  measurement  of  performance.  The  keys 
to  arriving  at  the  doorstep  of  “world  class” 
status  are  engendered  in  integrating  the 
twenty  concepts  listed  above,  with 
organizational  and  planning  characteristics. 

The  enclosed  techniques  represent 
individual  performance  measurement— which 
can  be  used  in  a  stand-alone  approach  or 
selectively  aggregated  into  families  of 
performance  indicators.  They  cover  methods 
for  measuring  program  performance, 
financial  performance,  operational 


performance,  and  project  performance, 
among  others. 

Do  not  be  distressed  if,  when  reading  the 
enclosed  material,  it  seems  to  have  no 
apparent  connection  with  what  is  currently 
thought  to  be  performance  measurement.  It 
does.  The  forces  governing  the  operation  of 
organizations  have  dramatically  changed  in 
recent  years  and  current  COE  practices  have 
not  yet  caught  up.  Read— no,  study— the 
material  and,  after  a  short  period  of 
reflection,  you  will  begin  to  make  some  sense 
of  it  all  and  begin  applying  the  tools  and 
methods  discussed.  They  are  powerful  and, 
as  research  has  shown,  are  being  employed 
by  “cutting  edge”  organizations  as  we  move 
into  the  21'st  century. 

With  these  ideas  in  mind,  let  us  embark 
on  a  journey  of  performance  measurement 
and  customer  driven  quality  outcomes.  The 
journey  is  both  humbling  and  awakening. 
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I.  INTRODUCTION 


J  J  J  J  J 


Recent  Congressional  findings 
have  demonstrated  that  there  is 
considerable  waste  and 
inefficiency  in  Federal  programs  which  have 
resulted  in  the  loss  of  confidence  of  the 
American  people  in  the  Government.  This 
has  reduced  the  Federal  Government's  ability 
to  address  critical  public  concerns.  Federal 
managers  are  considerably  restricted  in  their 
efforts  to  improve  program  efficiency  and 
effectiveness  because  of  inadequate  strategic 
planning  and  information  on  program 
performance.  In  addition,  congressional 
policy  making,  appropriation  decisions,  and 
program  oversight  are  severely  hampered  by 
insufficient  attention  to  program  performance 
and  results.  In  order  to  overcome  these 
handicaps  a  number  of  public  laws  and 
Executive  Orders  were  promulgated  in  1993. 
They  have  turned  the  tide  on  the  way 
government  institutions  and  programs  are 
managed. 

The  Government  Performance  and 
Results  Act  (GPRA),  National  Performance 
Review  (NPR),  and  the  Executive  Order 
entitled,  Customer  Service  Standards  (CSS), 
were  all  initiated  in  1993.  In  addition,  the 
Malcolm  Baldrige  Award  Criteria  (as 
expressed  in  the  Army  Performance 
Improvement  Criteria,  APIC)  was  adopted  by 
the  Army  in  March  1995  for  the  ACOE  and 
the  Labor  Management  Partnership 
Agreement  was  promulgated  by  EO 12871  in 
1993.  Collectively,  they  resulted  in  an 
increased  emphasis  on  the  measurement  of 
government  performance  in  terms  of 


effectiveness  and  efficiency.  They  also 
established  a  new  foundation  for  strategic 
management  and  planning  including  the 
implementation  of  key  organizational  goals 
and  tactical  objectives  designed  around 
systems  and  continuous  process 
improvements.  There  is  a  renewed  emphasis 
on  customer  service  and  satisfaction. 
Resource  utilization  will  be  traceable  from 
the  input  stage  through  the  production 
process  to  the  output  and  results  stage.  The 
new  paradigm  will  re-engineer  the  way 
government  does  business  through  people, 
processes,  and  organization.  The  results  of 
these  actions  are  that  performance  measures 
and  benchmarking  will  link  the 
organizational  mission,  strategy,  goals,  and 
processes  of  government  output  and  outcome 
with  the  needs  and  desires  of  the  customer. 

The  Purpose  of  This  Report 

This  report  will  emphasize  performance 
measurement  tools  and  techniques  designed 
to  assist  those  involved  in  performance 
planning  and  measurement  activities.  This 
includes  all  elements  of  the  U.S.  Army  Corps 
of  Engineers  (COE),  the  office  of 
Management  and  Budget  (OMB),  and  the 
Government  Accounting  Office  (GAO). 
Practitioners,  Division  Performance 
Methodologists  and  District  Performance 
Specialist  will  find  it  particularly  useful.  It 
will  aid  in  deciding  what  and  how  to  measure 
performance  in  ways  generally  not  familiar  to 
those  currently  involved  in  the  process.  It 
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will  include  the  measurement  of  both 
particular  attributes  and  processes  using 
simple  measures,  single  and  composite 
indices,  control  charts,  statistical  process 
control  techniques,  and  benefit-to-cost 
analysis,  among  others.  The  more  complex 
methods  may  be  more  appropriately  adopted 
after  the  performance  measurement  program 
has  gained  widespread  acceptance  and 
applicability.  They  will  be  presented  here  for 
planning  purposes  and  potential  future  use. 
Performance  measurement  actually  consists 
of  at  least  six  components:  input 
measurement,  effectiveness,  efficiency, 
productivity,  output,  and  outcome. 

Inputs  are  the  resource  factors  that  go 
into  producing  a  product  or  service 
(Management  Concepts  Inc.,  1995).  They 
would  include  such  items  as  raw  materials, 
money,  time,  labor  costs,  knowledge, 
capital,  technology,  etc.  Effectiveness 
measures  whether  or  not,  or  to  what  degree, 
objectives  were  met  [Management  Concepts 
Inc.,  1995].  It  tests  whether  or  not  an 
organization  is  "doing  the  right  thing.” 
Productivity  is  the  relationship  between  the 
total  value  of  products  and  services  produced 
and  the  total  amount  of  resource  inputs  used 
to  produce  them  [Certo,  1989].  It  is 
basically  an  index  of  output  divided  by  input 
and  typically  measures  cost  per  unit  of 
output.  Efficiency  is  the  ratio  of  actual 
output  to  effective  organizational  capacity 
[Stevenson,  1993].  It  determines  if  an 
organization  is  "doing  things  right Output 
is  the  amount  or  value  of  products  or  services 
created  by  the  production  process 
[Management  Concepts  Inc.,  1995],  Finally, 
outcome  (in  a  COE  context)  is  the  fulfillment 
of  customer  driven  needs  and  desires  by 


making  a  completed  engineering  structure 
operational  and  able  to  perform  its  planned 
purpose  [Management  Concepts  Inc.,  1995]. 
A  simplified  example  will  help  explain  the 
differences  between  these  concepts. 

Branch  "A"  completed  eight  water 
resources  reports  during  the  last  FY 
expending  $3,700,000  in  total.  Comparable 
branches  throughout  the  COE  have  shown  a 
capacity  to  complete  12  similar  reports 
(effective  capacity)  in  the  same  time  frame 
using  the  same  resource  inputs.  The 
management  in  Branch  "A"  originally  set  a 
goal  to  produce  10  reports  during  the  FY,  up 
from  9  the  previous  year.  The  branch 
employees  30  FTE'S. 

In  this  example,  resource  inputs 
amounted  to  $3,700,000  which  consisted  of 
the  total  cost  of  doing  the  work  for  that  year. 
It  includes  labor,  overhead,  rents,  utilities, 
materials,  supplies,  etc.  The  Branch's 
productivity  amounted  to:  $3,700,000/30 
FTE's  or  $123,333  expended  per  FTE 
(output  per  person).  Actual  output  was  8 
reports.  The  organization's  efficiency  was: 
8  actual  reports/  12  report  capability  or 
66.7%.  Effectiveness  amounted  to:  8  actual 
reports/ 10  report  goal  or  80% .  The  outcome 
produced  delays  in  satisfying  customer  needs 
stemming  from  4  reports  which  did  not  get 
completed  but  could  have.  The  outcome  or 
results  may  be  far  more  complicated  and 
involved,  but  this  is  one  of  the  immediate 
consequences  of  not  operating  efficiently  or 
effectively.  A  close  examination  of  this 
anecdote  will  clarify  the  value  of  each  of  the 
performance  measurement  tools  discussed. 
These  concepts  will  be  covered  in  more  detail 
later  in  the  report. 
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Prior  to  delving  into  the  specifics  of 
tools  and  techniques,  a  model  organizational 
business  framework  will  be  discussed  as  a 
reference  and  to  set  the  stage  for  performance 
measurement.  This  will  include  brief 
discussions  related  to  strategic  management, 
setting  goals  and  objectives,  tactical  planning 
and  business  operations  and  the  production 
process.  This  background  is  necessary  so 
that  users  can  start  measurement  formulation 
and  evaluation  with  an  idealized  model  in 
mind.  The  connection  between  mission, 
programs,  processes,  output  and  outcome, 
and  measurement  must  be  made  so  as  not  to 
lose  sight  of  why  this  effort  is  being  done  in 
the  first  place. 

Benefits  Of  Performance  Measurement 
Programs 

There  are  many  reasons  to  measure 
organizational  performance  but  four  are 
particularly  outstanding  and  appropriate  for 
today's  total  quality  management  perspective. 
Performance  measurement  forces  institutions 
and  people  to  focus  on  real  as  opposed  to 
imaginary  goals. 

The  first  is  in  aiding  and  satisfying 
customer  needs  and  wants.  This  may  be  the 
single  most  important  mission.  Institutions 
frequently  tout  customer  satisfaction.  The 
trouble  is  that  proclamations  often  serve  as 
advertising  gimmicks  rather  than  realities.  By 
measuring  customer  satisfaction,  the 
organization  is  compelled  by  survey  results 
to  behave  as  they  advertise.  It  is  actual 
performance  that  brings  customers  back. 
Word  of  mouth  is  a  very  powerful 
advertising  scheme.  The  definition  of  a 
customer  should  reflect  both  external 


customers  (tax  payers,  partners, 
stakeholders,  etc.)  but  should  include  internal 
customers  (other  branches,  divisions,  etc.)  as 
well. 

Performance  measurement  also  allows 
an  organization  to  monitor  progress.  It 
directs  management  to  examine  corporate 
systems  and  processes  and  leads  to  a 
continuous  improvement  ethos.  It  prevents 
activity  fixation  and  results  in  activity 
tracking-and  elimination  by  reducing  the 
number  of  process  step  required  to  complete 
a  task  or  work  endeavor.  This  in  turn 
reduces  costs  and  increases  the  speed  with 
which  output  is  accomplished. 

Benchmarking  is  the  third  reason  for 
measuring  performance.  Benchmarking  is 
essentially  comparing  your  business  practices 
and  facts  against  the  “best  in  the  business”. 
It  is  a  standard  of  reference  to  drive  change 
and  foster  competition.  Management  by  facts 
is  growing  in  use  and  this  allows  for  an 
objective  comparison  between  one 
organization's  practices  and  another's.  It  is 
these  comparative  differences  that  focuses 
attention  on  problems  and  explains  why  one 
organization  is  out-performing  another. 

The  fourth  major  reason  to  measure 
performance  lies  in  driving  change. 
Monopolies,  for  example,  have  no  real 
competitors  and  as  a  result  are  not  price 
competitive  nor  innovative.  Product  and 
service  price,  quality,  product  differentiation, 
flexibility,  and  timeliness  are  not 
characteristic  of  monopoly  operations. 
Customers  have  no  options  or  choices. 
Performance  measurement  drives  change  and 
tends  to  prevent  organizational  stagnation. 
The  value  of  performance  "measurement" 
has  been  well  stated  in  the  literature: 
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"If  you  do  not  have  the  components 
of  excellence— statistical  process 
control,  continuous  improvement, 
benchmarking,  the  constant  pursuit 
of  excellence,  the  capability  of 


knowing  how  to  do  the  right  thing 
the  first  time— then  you  don't  even 
get  to  play  the  game "  [Barker,  1992J. 
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II.  ORGANIZATIONAL 
PLANNING 


Planning  is  problem  resolution  for 
the  future  in  a  dynamic 
environment  [Hogans,  1991].  It 
describes  what  you  intend  to  accomplish: 
vision,  mission,  goals,  and  objectives.  It 
also  states  the  how,  when,  where  stated 
within  the  plan.  The  purpose  of  an 
organization  is  first  expressed  as  a 
directional  vision  statement,  accompanied  by 
a  general  proclamation  of  mission,  and 
finally  delineated  in  terms  of  goals.  These 
goals  are  more  specifically  defined  by 
objectives.  The  plan's  effect  is  to  steer  the 
organization  and  all  its  elements  to  move  in 
the  same,  common  direction;  to  focus  effort. 

The  major  plans  of  action  to  achieve 
vision,  mission,  goals,  and  objectives  are 
called  strategies.  Strategies  represent  the 
interests  of  top  management.  They  are 
chiefly  concerned  with  long  term  activities 
designed  to  achieve  objectives,  but  may  also 
be  directed  at  major  short  term  actions. 
Strategic  plans  focus  on  three  fundamental 
questions:  (1)  Where  are  we  now?  (2)  Where 
do  we  want  to  go?  (3)  and,  How  do  we  get 
there? 

Intermediate  (tactical)  plans  and 
operational  plans  are  concerned  with  the 
achievement  of  less  significant  goals  and 
objectives  and  are  typically  the  responsibility 
of  middle  and  lower  management.  A  tactical 
plan's  prime  purpose  is  directed  at 
accomplishing  objectives  between  strategies 
and  operational  plans.  This  function  can  be 
associated  with  regional  Division  offices  in 
the  Corps.  Usually  middle  and  lower 


management  efforts  are  trained  on  fulfilling 
upper  management  vision,  mission,  goal  and 
objectives  using  more  specific  planning 
statements.  Middle  and  lower  management 
direct  much  of  their  energy  at  implementing 
strategic  plans.  Tactical  plans  portray 
intermediate  time  horizons. 

Operational  plans  deal  with  day-to-day 
work  efforts  in  time  frames  usually  less  than 
a  year.  They  are  the  principle  arena  of  lower 
management  and  commit  much  less  resources 
than  strategic  and  intermediate  plans. 
Performance  measurement  is  the  cornerstone 
of  activities  at  this  level  of  planning  [Hogans, 
1991].  Managers  are  preoccupied  with 
implementation  of  higher  level  goals  and 
objectives.  They  are  also  concerned  with  the 
most  efficient  and  effective  use  of  the 
organization's  resources  through  intermediate 
and  operational  planning.  These  functions 
are  most  indicative  of  District  offices  in  the 
COE.  Note  that  the  delegation  of 
accountability  between  the  various  levels  of 
strategic,  tactical,  and  operational  planning 
and  activities  will  vary  depending  upon  the 
organizational  structure.  Figure  1  shows 
these  interrelationships.  The  time  horizon  of 
strategic  planning  is  usually  one  to  five 
years;  that  for  tactical  planning  is  between 
two  months  and  two  years,  and  for 
operational  planning  is  a  year  or  less.  The 
time  frames  will  vary  from  organization  to 
organization  but  these  represent  the  generally 
accepted  industry-wide  norms  [Higgens, 
1991].  As  change  occurs  more  rapidly,  the 
need  for  organizational  planning  requires 


Organizational  Planning  -  5 


increased  emphasis.  This  is  further  amplified 
during  periods  of  delayering,  and 
reorganization,  and  re-engineering  when 
matters  become  increasingly  difficult  and 
complex.  It  is  during  vexing  times  that 
performance  measurement  becomes  most 
valued  because  it  helps  maintain  direction 
and  focus. 

Plans  are  frequently  classified  into  two 
basic  types:  standing  plans  or  single  use 
plans  [Hogans,  1991],  These  are  also 
depicted  in  Figure  1.  Both  are  apparent  in 
most  organizations  in  one  form  or  another. 
Standing  plans  are  used  to  guide 
organizational  activities  which  recur  over 
time.  They  are  driven  by  the  use  of  policies, 
procedures,  and  regulations.  These  tend  to 
instill  discipline  and  keep  the  organization  on 
track  over  extended  periods  of  time.  They 
prevent  or  minimize  drift  in  goals  and 
objectives.  Examples  of  these  might  include 
Principles  And  Standards,  Equal 
Employment  Opportunity  (EEO)  guidance, 
Policy  Digest  and  the  numerous  Engineering 
Regulations  (ER's)  which  have  had  staying 
power  over  time. 

Single-use  plans  are  used  once  to 
accomplish  a  specific  purpose  and  discarded. 
They  may  last  a  few  days  or  ten  years. 
Single  use  plans  are  of  three  type;  strategic, 
intermediate  (tactical),  and  operational  plans. 
These  are  depicted  in  Figure  1  as  well. 

Strategic  plans  are  the  broad  based  plans 
requiring  the  extensive  use  of  resources  and, 
as  described  earlier,  are  aimed  at  achieving 
the  major  goals  and  objectives  of  an 
organization.  They  would  include  vision  and 


mission  plans  and  longer  term  agency 
activities  such  as  reorganization,  performance 
measurement,  performance  plan 
development,  etc. 

Intermediate  plans  help  convert 
strategies  into  operational  activities.  They 
include  Divisional  plans,  specific  flood 
control  and  navigation  related  programs,  and 
complex  projects.  Examples  would  include 
a  particular  three  year  Feasibility  report,  the 
development  of  an  open  ended  contract,  etc. 

Operational  plans,  on  the  other  hand, 
focus  on  day-to-day,  week-to-week,  or 
monthly  activities  for  a  period  of  a  year. 
They  would  include  annual  budgeting, 
meeting  internal  and  external  milestones  for 
specified  reconnaissance  reports,  sub- 
products  for  Feasibility  Reports,  Limited 
Re-evaluation  Reports  (LRR),  General 
Reevaluation  Reports  (GRR),  and  personnel 
training,  to  mention  a  few.  Operational  plans 
are  concerned  chiefly  with  the  production  or 
transformation  process. 

Figure  2  shows  a  diagram  depicting  the 
above  relationships.  The  diagram  shows  the 
characteristics  associated  with  each  level  of 
management.  The  size  of  the  rectangles 
represent  the  magnitude  of  responsibility  for 
each  management  level.  For  example,  upper 
management  emphasizes  long  term  planning 
of  a  strategic  nature  and  focuses  mostly  on 
vision,  mission,  goals  and  objectives. 
Middle  management  is  mostly  energized  on 
issues  related  to  objective  fulfillment,  and 
lower  management  emphasizes  rules  and 
regulations. 
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Figure  1  -  Process  Structure  and  Flow 
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Figure  2  -  How  Each  Management  Spends  Time 
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III.  PERFORMANCE 
MEASUREMENT  AND 
PLANNING  LINKAGES 
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The  above  narrative  describes  the 
basic  business  environment 
performance  measurement  is 
designed  to  evaluate.  It  is  this  theoretical  or 
model  framework  that  actual  performance  is 
to  be  gaged  against.  If  vision,  mission, 
goals,  and  objectives  conform  to  strategic, 
tactical  and  operational  plans  then  a  link  has 
been  made  that  will  allow  organizational 
performance  to  be  at  its  peak.  Imperfect 
resource  use  will  cause  an  adverse  impact  or 
distortion  in  this  idealized  system.  It  is  often 
characterized  as  waste,  inefficiency,  rework, 
duplication  of  effort,  and  related  anomalies. 
This  distortion  is  detectable  in  misallocated 
resource  inputs,  yielding  inefficiency,  low 
productivity,  ineffectiveness,  reduced  output, 
and  the  wrong  results.  Performance 
measurement  quantifies  and  helps  diagnose 
the  departure  of  actual  system  performance 
from  model  or  optimum  system  performance. 
According  to  such  process  experts  as  Edward 
Deming  and  Joseph  Juran,  management  is  80 
percent  responsible  for  poor  organizational 
performance  problems  while  employees  are 
responsible  for  the  remaining  20  percent 
[Deming,  1982]. 

Results  represent  the  impact  of  the 
organization  on  its  customers.  It  is  often 
difficult  to  measure  results  as  they  may  take 
years  to  accrue.  For  example,  a  flood 
control  plan  may  call  for  complete  protection 
against  a  fifty  year  storm.  A  fifty  year  storm 
can  be  expected  to  occur,  on  the  average, 
once  every  fifty  years  (a  0.02  probability  in 
any  one  year).  This  means  that  the  real 


outcome  or  results  can  only  be  tested  twice  a 
century.  Similar  dilemmas  exist  for 
navigation,  recreation,  environmental 
assessment,  water  supply,  hydropower,  and 
other  COE  missions.  Consequently, 
outcomes  may  not  be  known  for  long  periods 
of  time;  but  we  can  readily  measure  output. 
As  previously  stated,  GPRA  defines  outcome 
as  "an  assessment  of  the  results  of  a  program 
as  compared  to  its  intended  purpose". 
Outcomes  can  be  short  term,  intermediate 
term,  or  long  term  measures  of  performance. 
One  of  the  challenges  of  GPRA  and  APIC  is 
to  determine  how  to  measure  those 
performance  attributes  that  are  long  term. 
Outcomes  are  plagued  by  the  fact  that  they 
must  be  realized  (actually  happen)  to  be 
measured  or  knowable.  Output  is  generally 
predictable  and  can  realized  in  the  near  term. 
This  assumes  that  projects  do  not  have 
significant,  construction  additions  or 
increments  added  in  the  future.  It  is  in  the 
confirmation  of  "intended  purpose"  that 
makes  outcome  difficult  to  verify.  Analytical 
modeling  and  evaluation  are  simulations  or 
emulations  of  expected  results  and  not 
confirmation  of  those  results. 

It  needs  to  be  stated  at  the  outset  that 
performance  measurement  has  its 
weaknesses.  For  one,  performance  cannot 
usually  be  fully  described  by  a  single 
measure;  it  is  a  complex  phenomenon. 
Secondly,  risks  and  uncertainty  are  involved 
in  projecting  the  future  (part  of  the  planning 
process).  Risks  can  be  accounted  for  by 
contingency  planning  and  by  statistical 
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estimation.  Uncertainty,  on  the  other  hand, 
is  the  real  unknown  and  is  not  quantifiable  by 
definition.  It  represents  the  real  threat  to 
performance  and  planning.  The  best  defense 


against  uncertainty  is  to  include  intermediate 
term  planning  to  make  mid  point  course 
corrections  possible.  This  is  one  of  the 
reasons  plans  need  to  be  updated. 
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IV.  IDENTIFYING 
PERFORMANCE 
MEASURES 


In  order  for  performance 

measurement  to  be  successful, 
several  questions  need  to  be 
addressed.  For  example,  what  type  of 
measure  is  it?  What  is  the  time  frame  for 
measurement?  How  often  should 
measurements  be  taken?  And,  how  plausible 
is  implementation?  These  are  important 
questions  since  such  things  as  seasonality  can 
have  an  impact  on  what  is  measured,  the 
units  of  measure  can  differ  for  the  same 
attribute,  and  what  we  intend  to  measure 
may  not  be  controllable  by  the  agency. 
Some  attributes  may  be  aesthetically 
desirable  but  may  be  useless  in  measuring 
performance. 

There  are  a  number  of  factors  which 
need  to  be  examined  when  doing 
performance  measurement.  It  is  necessary  to 
know  what  resource  inputs  are  being  used,  a 
map  of  processes  and  systems  that  are  at 
work,  the  outputs  expected,  the  outcomes 
anticipated,  the  ambient  efficiency  of 
operations,  and  the  existing  effectiveness  of 
on-going  operations.  These  all  represent 
baseline  conditions  from  which 
improvements  can  be  measured.  Performance 
measurement  requires  knowledge  of  all  these 
characteristics  to  be  fully  useful.  The  user 
must  also  be  cognizant  of  the  time  frames  of 
measurements:  short  run,  mid-term  and  long 
term  events. 

“Ideal”  performance  measures  would 
satisfy  fourteen  criteria.  They: 


□  Must  communicate  the  progress  towards 
satisfying  vision,  mission,  program,  key 
goals  and  objectives, 

□  Should  be  quantifiable  to  maintain 
objectivity, 

□  Must  assign  accountability  in  carrying 
out  the  measurement  process, 

□  Must  be  acceptable  to  the  organization, 

□  Must  be  credible  and  believable, 

□  Must  be  timely, 

□  Must  be  reliable  and  verifiable, 

□  Must  be  effective,  have  impact,  and 
steer  the  organization  toward  improved 
strategy  and  performance, 

□  Should  reflect  constancy  of  purpose  and 
continuous  process  improvement, 

□  Must  be  comparable  to  other  measures 
for  benchmarking  purposes, 

□  Must  be  applicable  and  substantive, 

□  Should  be  used  in  conjunction  with 
other  measures, (one  measure  does  not 
usually  tell  the  story)  and 

□  Must  be  understandable  by  those  using 
and  reviewing  them. 

□  Must  reflect  variables  over  which  there 
is  at  least  some  control. 

The  above  criteria  will  aid  in  leading  toward 
the  development  of  effective  performance 
measurement.  It  will  not  always  be  possible 
to  satisfy  all  criteria.  It  will  take  several 
years  of  trial,  error,  and  screening  to  find 
the  measures  appropriate  for  any  particular 
organization.  It  is  more  likely  that,  as 
customer's  needs  change,  performance 
measures  will  follow  suit. 
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V.  SYSTEMS 
AND  PROCESSES 


J  J  J  Q 


Performance  measurement  is 

strongly  tied  to  systems  and 
processes.  All  work  emulates  a 
process.  Loosely  defined,  a  system  is  a 
configuration  of  things  or  processes  so 
inter-connected  as  to  behave  as  one 
[Certo,1989].  A  system  is  not  possible 
without  processes.  Processes  indicate  some 
sort  of  work  activity  is  taking  place  toward  a 
desirable  objective.  Systems  are  made  of 
processes  working  together  in  series  or  in 
parallel.  In  the  context  of  business 
operations,  resource  inputs  go  through 
multiple  processes  called  a  transformation  to 
produce  outputs  which  have  outcome  targets. 
A  COE  District  may  be  thought  of  as  a 
system  with  each  Division  acting  as  a 
separate  process  and  each  Branch  as  a 
subprocess.  In  the  simplest  case  a  system 
and  a  process  may  be  one  in-the-same.  In 
measuring  the  performance  of  systems  and 
processes,  one  vital  activity  that  must  take 
place  is  system  and  process  mapping.  A 
process  map  is  a  flow  chart  of  events  and 
activities  that  take  place  during  the 
transformation  process.  It  is  not  possible  to 
measure  performance  of  a  system  or 
processes  and  make  improvements  unless  the 
work  activity  steps  are  known.  Mapping 
starts  with  a  flow  diagram  of  ambient  or  pre¬ 
improvement  processes.  Alternative  systems 
and  processes  are  also  mapped  on  a  flow 
diagram  for  comparative  performance 
characteristics.  Improvements  represent 
decreases  in  the  total  number  of  tasks  or 
improvements  in  the  quality  of  the  existing 


activities  within  the  system  or  process.  This 
information  will  be  useful  later  in  the  paper 
when  a  distinction  is  made  between  the 
performance  measurements  of  attributes  and 
those  of  processes. 

Figure  3  shows  the  relationship  between 
a  system  and  a  process.  This  example  is 
very  simple  and  is  being  used  for  illustration 
purposes.  A  system  is  the  sum  of  its 
processes.  There  may  be  many  processes 
comprising  a  system  and  in  large 
organizations  there  usually  are  many  sub 
processes  which  coalesce  to  form  major 
process.  The  processes  of  interest  are  called 
"core"  processes.  These  are  the  activities 
which  are  pivotal  to  an  organization's 
strategic  operations.  They  are  the  "jugular" 
central  to  mission  accomplishments.  Notice 
that  the  system  depicted  consists  of  four 
processes  with  each  composed  of  an  input,  the 
production  process  itself  (transformation),  an 
output,  and  a  feedback  loop.  The  feedback 
loop  is  designed  to  correct  or  modify  the 
input- transformation,  process-output 

sequence  as  needed.  This  feedback  could  be 
due  to  customer  comments  concerning  the 
quality  of  output.  Notice  that  processes  (1) 
and  (4)  are  independent  and  complete  in  and 
of  themselves.  Processes  (2)  and  (3)  show 
inputs  combining  into  the  same  process  to 
produce  an  output.  The  dotted  lines  represent 
the  boundaries  of  each  process;  those  work 
endeavors  which  are  indigenous  to  that 
process.  Such  relationships  are  possible  and 
frequently  found  in  systems  analysis.  Good 
test  points  for  process  performance 
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Figure  3  -  Systems  and  Processes 


measurement  are  at  boundaries  clearly 
marked  at  the  beginning/end  of  a  process 
(interface  points).  It  should  be  emphasized 
that  measuring  output  alone  may  be  too  late 
in  the  sequence  of  events.  Performance 
measurement  is  designed  to  capture  problems 
before  they  wind  up  in  output.  Quality  is  to 
be  designed  into  the  process;  not  inspected 
out.  One  example  of  this  in  COE  operations 
is  when  hydraulic  and  hydrology  data  are 
combined  with  the  economic  data  in  a  flood 
control  study  through  computer  modeling  to 
produce  average  annual  damages  for  an 
existing  or  future  without  condition  channel 


configuration.  The  results  of  this  are  then 
fed  into  another  process  (cost  analysis)  to 
determine  a  benefit-to-cost  ratio.  The  system 
analysis  can  go  on  until  the  report  is  written 
or  the  project  is  constructed. 

A  good  test  to  determine  if  an  action  is 
corrective  or  is  just  a  quick  fix  is  to  observe 
if  that  action  is  directed  at  the  output  of  a 
process  or  at  the  way  the  process  operates. 
The  former  is  a  quick  fix;  the  latter,  a 
process  improvement.  One  is  corrective  and 
the  other  is  an  organizational  drug. 

When  a  system  or  process  is  not 
working  at  peak  performance  this  is  called 
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process  dysfunction.  When  they  are  not 
working  a  all,  it  is  called  system  or  process 
failure.  Failures  sometimes  happen  along  an 
assembly  line  causing  a  shut-down  of  a 


production  process.  Dysfunction  is  far  more 
common  because  it  represents  systems  and 
processes  not  operating  in  the  most  efficient 
and  effective  manner. 
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VI.  ORGANIZATIONAL 
STRUCTURE  AND 
PROCESS  MANAGEMENT 


The  previous  material  was 
designed  to  acquaint  the  reader 
with  standard  business  structure 
and  processes.  The  performance  analyst 
needs  to  be  aware  of  what  it  is  that  he/she  is 
really  trying  to  measure  and  the  complexities 
involved.  System  and  processes  are  replete 
with  many  levels  of  management  making  the 
whole  effort  difficult  to  measure  and  correct. 
Organizational  structure  can  dictate 
performance.  The  traditional  vertical, 
command-and-control  business  structure  is 
being  attacked  today  as  a  leading  cause  of 
inefficiency  and  ineffectiveness.  It  defeats 
the  purpose  of  performance  improvements 
with  its  many  disconnects  and  hand-offs 
when  work  passes  from  one  branch  or 
division  to  another.  The  horizontal  structure 
is  gaining  in  popularity  because  it  is  less 
autocratic  and  less  bureaucratic.  It  also 
allows  for  empowerment  of  employees.  The 
most  recent  change  in  favor  of  the  horizontal 
structure  is  the  new  employee-to-management 
ratio  criteria.  It  is  designed  to  eliminate  or  at 
least  reduce  micro  management  and 
over-control  authority.  One  of  the  keys  of 
performance  improvements  is  not  in  just 
doing  things  right  or  doing  the  right  things, 
but  is  simply  aimed  at  getting  rid  of  what 
does  not  need  to  be  done  in  the  first  place 
[Drucker,  1991],  An  examination  of  Figure  4 
shows  how  structure  can  impact  performance 
of  work  activity.  In  the  vertical,  pyramid 
business  structure,  in  order  for  an  activity  in 
office  l)to  reach  office  7  it  must  traverse  (see 
darkened  line)  the  chain  of  command  through 


offices  2,  3,  4,  5,  and  6.  This  is  not  the  case 
with  the  horizontal  structure  shown  below  it. 
There  are  two  fewer  steps  in  the  flat 
organization  even  though  21  offices  exist  in 
both  structures.  This  savings  spread  out  over 
a  year  in  a  typical  business  environment  can 
amount  to  substantial  savings  in  delivery  time 
to  customers  and  in  corporate  operating 
expenses.  The  horizontal  structure  is  less 
complex  and  naturally  conducive  to 
effectiveness  and  efficiency  gains.  Not  only 
is  there  an  absolute  reduction  in  the  number 
of  production  activities  or  steps,  but  the 
cross-functional  complications  are  reduced. 
Territorial  disputes,  power  structures,  and 
"gaming"  all  decline.  Even  more 
enlightening  would  be  the  cross-functional 
capability  of  moving  from  office  1  to  7 
directly  as  with  self-managed  teams.  The 
number  of  layers  of  management  are  reduced 
as  well,  effectively  reducing  the  dividing 
aspects  of  corporate  fiefdoms.  Waste, 
errors,  redundancy,  and  rework  naturally 
decline. 

Figure  5  captures  the  gist  of  what  it  is 
that  is  being  analyzed.  If  the  system  and  its 
processes  are  performing  smoothly,  then  all 
the  performance  measures  will  produce 
desirable  results.  Performance  measures  can 
quantify  the  driving  characteristics  at  any 
point  in  the  input-transformation-output— 
outcome  sequence.  This  quantitative 
feedback  is  what  management  needs  to  act 
upon.  The  process  is  sending  a  cause  and 
effect  signal  to  whoever  is  willing  to  measure 
and  act  on  it. 
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Figure  4  -  Organizational  Structure  and  Performance 
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Figure  5  -  Systems  and  Processes 


The  major  problem  in  the  past  has  been  that 
no  one  was  responsible  or  accountable  for 
knowing  and  measuring  the  entire  process. 
The  private  sector  has  as  part  of  their 
corporate  structures  “operations 
departments”  for  this  purpose.  In  addition, 
process  mapping  is  tedious  and  difficult. 
Attention  in  the  past  has  been  focused  on 
specialities  within  the  system  (Divisions, 
Branches,  Sections,  etc.)  with  little,  if  any, 
comprehensive,  system  orientation. 
Performance  measurement  changes  that. 

Figure  5  also  shows  examples  of 
resource  inputs  which  go  into  the 
transformation  system  to  produce  outputs  and 
outcomes.  As  stated  earlier,  measurement  of 
results  is  a  measurement  taken  too  late. 


Resources  have  been  expended,  waste 
accumulated,  redundancies  completed,  and 
rework  waiting  around  the  corner. 
Measurement  must  be  taken  at  critical  points 
along  the  process  and  of  the  process  itself. 
Critical  locations  will  depend  upon  the  actual 
process  being  mapped.  Once  the  problems 
with  performance  have  been  resolved  and  the 
solutions  engineered  in,  then  performance  of 
processes  and  subprocesses  can  be  reduced  to 
periodic  checks.  The  tools  and  techniques 
provided  in  the  following  paragraphs  will 
enable  management  to  quantify  the  adverse 
characteristics  of  various  organizational 
structures  and  compare  one  structure  with 
another  for  performance  improvements. 
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VII.  PERFORMANCE 
MEASUREMENT  AND 
KEY  RESULT  AREAS 


The  use  of  Key  Result  Areas 
(KRA's)  is  a  classic  case  of 
doing  more  with  less.  The  most 
important  concern  for  determining  corporate 
thrust  is  to  concentrate  the  resources  and 
energy  of  the  organization  on  a  few  KRA's 
with  the  highest  return  [Albrecht,  1985].  It 
is  not  necessary  to  tackle  all  problem  areas. 
This  is  called  leveraging.  It  pays  dividends 
to  concentrate  on  those  problems  and  take 
advantage  of  opportunities  which  are  most 
significant.  Concentrate  on  objectives  which 
are  likely  to  have  high  payoff.  Key  leverage 
points  found  in  “best  practice”  organizations 
are  spotlighted,  for  example,  by  five  KRA's: 
customer  satisfaction,  internal  and  external 
supplier  effectiveness,  program  management, 
work-life  quality,  and  cost  per  unit. 
Customer  satisfaction  is  driven  by  product 
quality  and  customer  perception.  Supplier 
effectiveness  means  delivering  a  quality 
product  or  service  on-time  and  within  cost. 
The  theme  of  program  management  is 
completion  as  scheduled  and  budget 
execution.  Work  life  quality  is  connected 
with  safety,  moral,  training,  and  EEO/AA 
accomplishments.  Cost  per  unit  is  driven  by 
dollars  and  full  time  employee  (FTE)  inputs. 
The  type  of  KRA  chosen  will  depend  upon 
the  organization  and  its  goals  and  objectives. 

Meaning  of  Critical,  Core  or  Key 
Processes 

Whenever  someone  speaks  of  key  core 
or  critical  processes,  they  are  usually 


alluding  to  some  basic  or  fundamental 
transformation  system  that  leads  to  getting 
things  done  in  the  least  costly,  most  effective 
manner.  Waste  minimization  is  described  by 
waiting  time,  review  time,  rework  and  transit 
time.  Management  is  often  added  to  this  list. 
Value-added  is  maximized  and  cycle-time  is 
as  short  as  practicable.  Let  us  take  a  little 
mystery  out  of  the  notions  of  core  processes 
and  critical  processes  by  examining  the 
problem  in  the  form  of  a  maze  as  depicted  in 
Figure  6.  The  resource  input  areas  contain 
the  human,  equipment,  land,  leadership, 
resources  needed  to  carry  out  a  mission.  The 
maze  portion  is  really  a  depiction  of  the 
transformation  processes.  It  is  the  heart  of 
the  subject  at  hand.  The  output/outcome 
represent  the  results  in  terms  of  products  and 
services  rendered  by  the  this  process. 

Over  time,  the  traditional  pyramid  or 
vertical  organization  become  increasingly 
more  bureaucratic,  layered,  regulation- 
riddled,  and  cycle-time/non- value-added 
crippled.  The  older  the  organization,  the 
more  likely  this  will  be  observed.  Figure  6 
begins  to  look  familiar  and  descriptive  of 
what  the  production  or  transformation 
process  resembles.  It  is  complex,  appears  to 
lack  organization  and  a  sense  of  direction,  is 
stove  pipe  oriented,  has  many  dead-ends  and 
unnecessary  paths,  and  often  grows  even 
more  complex  over  time.  It  may  also  have 
more  than  one  production  path  from  the  input 
stage  to  the  output  stage  thereby  increasing 
confusion.  The  complexity  actually  resembles 
a  three  dimensional  maze  rather  than  the  one 
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dimensional  maze  depicted.  This  is  what  the 
modern,  Taylorist  (command  and  control) 
organization  resembles;  a  complex, 
frustrating  maze  without  a  map-the- 
production  battleground. 

The  search  for  critical  or  core  processes 
is  a  search  through  such  mazes  to  find  how  to 
accomplish  work  in  the  most  straightforward, 
simple  process  possible.  This  is  the  lost  road 
in  today’s  non-TQM  organizations.  It  also 
explains  the  new  emphasis  on  systems  and 
process  mapping  (PERT  or  network 


analysis),  cycle-time,  value-added,  waste 
analysis,  critical  process  identification,  etc. 
Figure  7  shows  what  the  objective  of  all  the 
previously  discussed  NPR,  GPRA,  and  APIC 
requirements  are  about;  reengineer,  realign, 
restructure,  and  adopt  a  new,  simpler  and 
disciplined  business  process  roadmap 
reflecting  today’s  competitive  performance 
realities.  The  process  is  simplified,  traverses 
fewer  obstacles,  and  is  much  more  direct  in 
its  approach  to  output.  This  simple  maze  is 
the  core  or  critical  process  being  sought. 
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VIII.  THE  RISK  OF  USING 
SIMPLE  NUMBERS,  RATIOS 
AND  PERCENTAGES 


J  J  J  J  J 


The  use  of  raw  data  or  simple 
numbers  and  ratios  (percentages) 
is  a  frequently  used  form  of 
analyzing  business  operations.  What  most 
managers  and  decision-makers  do  not  know 
is  how  misleading  this  kind  of  data  can  be. 
It  often  leads  to  making  the  wrong  decisions 
for  the  wrong  reasons,  leaving  managers 
baffled  as  to  why  a  decision  they  made  a 
month  ago  did  not  produce  the  desired 
results.  There  are  a  number  of  reasons  for 
this.  They  include  [Wheeler,  1993]: 

1 .  First,  the  size  of  percent  differences 
between  two  numbers  compared 
over  time  is  directly  related  to  the 
base  of  the  numbers  themselves. 
For  example,  to  go  from  100  to  110 
points  represents  an  increase  of  10 
and  a  percent  change  of  10% .  To  go 
from  300  to  310  is  an  increase  of  10 
as  before  but  represents  a  3.3% 
increase.  Comparisons  can  be 
misleading  and  not  very  reliable 
because  they  do  not  take  into 
consideration  the  value  of  the  base 
number. 

2.  Period-to-period  changes  in  numbers 
can  be  caused  by  two  different 
forces.  One  is  due  to  the  natural 
variation  in  values  stemming  from 
random  fluctuations  that  affect  all 
measurements.  The  other  is  from 
changes  in  the  indigenous 
operational  process.  Yet,  in  a 


collection  of  data  assembled  over 
time,  great  attention  is  given  during 
executive  or  Board  meetings  to 
changes  in  values  as  measures  of 
performance  when,  in  fact,  these 
changes  are  random  and  may  not  be 
connected  with  operational  process 
performance,  equipment,  employee 
performance  or  management 
influence  at  all.  An  increased 
production  figure  is  believed  to  be  a 
positive  signal  that  things  are  going 
just  fine,  whereas,  a  decreased 
production  figure  signals  something 
is  amiss.  An  increasing  figure  may 
imply  that  things  are  going  well 
when,  in  fact,  there  are  underlying 
serious  problems  at  hand.  The 
increase  may  have  no  more  meaning 
than  the  hourly  fluctuation  in  daily 
temperature  readings  on  a  partly 
cloudy  day.  It  does  not  indicate  a 
pattern  or  trend  at  all.  If  action  is 
taken  to  “correct”  the  declining 
production  figure,  it  may  be  the 
wrong  thing  to  do  because  such  a 
change  was  random.  Thus,  acting 
on  simple  numbers,  ratios  and 
percent  changes,  generated  from 
one  period  to  another,  can  cause 
more  problems  than  it  solves.  A 
problem  is  then  created  which  did 
not  exist  before  causing  a 
snowballing  or  multiplier  effect  in  an 
otherwise  perfectly  good  production 
process.  Changes  in  values  are  not 
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necessarily  “signals”  that  something 
needs  to  be  done,  but  are  more  likely 
to  have  come  from  random  “noise”; 
something  not  to  be  acted  upon.  The 
diagnosis  and  detection  of  real 
process  or  performance  changes  is 
what  this  book  is  about.  This  does 
not  imply  that  the  use  of  such 
numbers  is  incorrect,  but  that  they 
must  be  used  in  the  proper  context 
and  supported  with  other  tools  of 
measurement. 

3.  When  examining  period-to-period 
changes  in  output  (or  any  attribute), 
the  usual  assumption  is  made  that 
whatever  happened  this  period  was 
related  to  things  that  happened 
during  the  period,  since  the  last 
reading.  This  assumption  may  also 
be  a  frequently  made  big  mistake. 
The  consequences  of  what  happened 
during  a  given  period  may  have  been 
cast  in  stone  three  or  four  periods 
earlier  and  its  effects  only  beginning 


to  show  up  now.  An  unusual  value 
in  the  past  may  have  caused  the 
unusual  value  currently  observed. 

4.  Simple  numbers  alone  (averages, 
percentages,  ratios)  only  “count”; 
they  do  not  convey  knowledge  about 
the  process  (how  the  count  value 
came  about).  There  are  no  clues  on 
how  to  correct  problems  but  only 
measure  results.  Not  every  change 
in  output  (either  up  or  down)  is  a 
signal.  Simple  numbers,  ratios  and 
percentages  are  valuable  assessment 
tools  when  used  with  other  process 
analysis  tools  such  as  PERT  charts, 
Shewhart  and  Demos  Control  charts, 
and  Quality  Functions  Deployment, 
among  others.  All  of  these  forms  of 
performance  measurement  will  be 
discussed.  The  reader  needs  to  be 
aware  of  some  of  the  potential 
failures  when  using  simple  numbers, 
ratios  and  percentages. 
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Univariate  Statistics 

Univariate  statistics— also  called  raw 
numbers-are  the  most  commonly  used  form 
of  presenting  performance  measurement  data. 
They  are  the  day-to-day  numbers  used  by 
nearly  everyone.  Their  value  lies  in  their 
simplicity  and  straightforwardness,  i.e., 
expenditures  grew  by  $2,500,000  this  year 
over  last  year.  This  is  a  clear  and 
understandable  statement.  Univariate 
statistics  tell  us  a  great  deal  about  what  we 
know  at  any  point  in  time  as  well  as  in  time 
series.  Their  weakness  lies  in  the  application 
of  the  numbers  in  the  proper  context,  i.e., 
grew  as  compared  to  what  or  whom.  They 
also  do  not  express  complex  concepts  or 
abstract  ideas  in-and-of-themselves  and  can 
be  misleading. 

Ratios 

A  ratio  is  a  simple  one  number  divided 
by  another.  It  can  be  turned  into  a 
percentage  by  being  multiplied  by  100.  It  is 
perhaps  the  second  most  commonly  used 
expression  of  organizational  performance, 
productivity  and  efficiency.  It  is  valuable  for 
expressing  the  quotient  relationship  at  a  point 
in  time  compared  to  other  ratios  or  is 
frequently  used  to  show  change  over  a  period 
of  time.  It  can  be  as  simple  as  dividing  one 
number  by  another  or  as  complex  as  dividing 
multiple  sets  of  mixed  units  of  numbers  by 
another  multiple  set  of  mixed  numbers.  It 


can  also  be  weighted  for  importance  effect. 
It  is  generally  easily  understood. 

Special  Purpose  Indexes 

Special  Purpose  Indexes  are  nearly  self- 
descriptive.  They  are  designed  to  have  a 
very  specific,  almost  unique  purpose.  They 
usually  require  weighting  based  on  some 
unique  criteria.  For  example,  an  important 
factor  in  TQM  today  is  Work  Life  Quality- 
an  encompassing  concept  designed  to  express 
worker  growth,  development,  and  general 
happiness  in  the  work  environment.  There 
are  no  known  single  indexes  which  measure 
this  attribute.  Thus,  analysts  generally  need 
to  “invent”  an  index  to  cover  their  specific 
organization  or  shop.  An  example  of  this 
will  be  given  later  in  the  text.  Special 
purpose  indexes  are  frequently  found  where 
attempts  are  being  made  to  quantify  or  find  a 
proxy  for  an  aesthetic  concept  or  an  amenity 
impact. 

Multifactor  Indexes 

Ordinarily,  productivity  indexes  examine 
the  relationship  of  one  number  or  value  to 
another.  Multifactor  indexes  take  into 
account  multiple  causes.  For  example, 
output  per  employee  is  a  common  single 
factor  index  of  performance  measurement. 
An  example  of  a  multifactor  index  is  output 
divided  by  the  number  of  employees  and 
amount  of  equipment  used— with  both 
numerator  and  denominator  expressed  in 
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dollars.  Here,  the  cause  of  output  is  being 
attributed  to  employee  input  and  equipment 
input.  In  organizations  which  are  very  labor 
intensive,  such  as  painters  and  carpenters,  the 
simple  ratio  of  output  to  labor  input  is  very 
appropriate.  In  organizations  which  are  both 
labor  and  capital  intensive,  performance  is 
better  measured  by  output  divided  by  the 
combined  value  of  labor  and  capital  input.  In 
turn,  where  capital  (equipment)  is  the  real 
producing  agent,  a  good  index  would  show 
output  divided  by  the  value  of  equipment; 
such  as  found  in  robotic  factories. 
Multifactor  indexes  are  very  useful  measures 
of  performance  and  can  be  used  to  examine 
existing  performance,  benchmarked 
performance,  or  be  used  to  observe 
performance  growth  over  a  span  of  time. 
While  a  single  measure  is  often  used  in  the 
numerator,  the  denominator  can  contain 
numerous  inputs  such  as  the  value  of  labor, 
equipment,  energy,  education,  etc.  used  to 
produce  the  output.  Multifactor  indexes  are 
not  always  easy  to  develop  and  may  be 
expensive  to  maintain  but  they  contain  a  good 
deal  of  valuable  information  as  to  the  cause 
agents  of  productivity  improvements  or 
declines.  They  will  be  covered  extensively 
in  this  text. 

Partial  Factor  Indexes 

Partial  factor  productivity  ratios  are 
usually  represented  by  dividing  the  value  of 
output  by  the  value  of  one  input  only.  For 
example,  expenditures  per  employee,  reports 
completed  per  engineer  errors  made  per 
drawing,  et  al.  As  alluded  to  above,  they  are 
most  useful  when  one  resource  input 
dominates  output.  Families  of  single  factor 


indexes  are  frequently  used  to  measure  the 
performance  of  an  organization,  office,  or 
shop.  They  are  frequently  easy  to  understand 
and  require  very  little  information. 

Simple  Average-of-Relative  Indexes 

This  index  is  a  simple  ratio  of  the  value 
of  a  performance  number  in  a  future  period 
of  some  attribute  divided  by  the  value  of  the 
same  attribute  in  a  base  year.  It  shows  the 
growth  or  decline  in  a  resource  or  output 
over  time.  The  common  anchor  is  the  base 
year.  This  ratio  is  usually  restricted  in  the 
sense  that  whatever  is  being  measured  in  both 
the  numerator  and  denominator  must  be  in 
the  same  units.  An  analyst  cannot  mix,  say, 
the  cubic  yards  of  concrete  divided  by  tons  of 
steel.  This  usually  has  no  meaning  unless 
someone  needs  to  know  the  relationship  of 
concrete  to  steel. 

Weighted  Aggregate  Index 

Weighted  aggregate  indexes  represent  a 
class  of  performance  measurement  techniques 
that  have  many  positive  attributes.  They 
allow  the  user  to  mix  units  of  measure, 
combine  seemingly  unlike  units  of  measure, 
weigh  the  relative  importance  of  a  mix  of 
resource  inputs  or  outputs  to  reflect  their 
relative  importance,  and  can  be  used  to 
benchmark  in  a  current  time  period  as  well  as 
be  employed  to  measure  or  monitor 
performance  over  time.  The  Consumer  Price 
Index  is  a  classic  example  of  this  index. 
Here,  a  mix  of  over  a  hundred  items  in 
mixed  units  (dollars  per  pound,  dollars  per 
liquid  ounce,  dollars  per  yard,  payments  per 
month,  et  al)  are  combined  to  form  one 
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measure  of  price  performance 
(inflation/deflation)  over  time.  This  may  be 
one  of  the  most  valuable  performance 
measurement  tools  the  COE  can  adapt.  It  is 
simple  to  use  and  extremely  flexible— 
especially  when  many  factors  needs  to  be 
considered  in  a  performance  index. 

Efficiency 

Efficiency  is  found  by  dividing  the 
actual  output  of  an  organization  by  its 
effective  capacity.  It  represents  how  well 
you  are  doing  relative  to  your  potential.  The 
user  needs  to  identify  the  units  of  measure 
and,  more  importantly,  define  effective 
capacity.  Effective  capacity  represents  the 
maximum  amount  of  output  given  a  process 
mix,  scheduling  problems,  equipment 
maintenance,  quality  parameters,  and  other 
environmental  factors.  This  may  seem  a 
little  evasive  to  capture  but  is  easily 
established  by  settling  on  a  base  or  pre¬ 
defined  set  of  circumstances  to  serve  as  the 
base  condition.  It  is  valuable  because  it 
shows  changes  in  capability  relative  to  the 
base  condition  over  time— is  the  organization 
becoming  more  or  less  efficient? 

Cycle  Time  Analysis 

No  other  parameter  may  be  more 
important  than  improving  cycle  time.  It  is  a 
significant  performance  indicator  and  will 
likely  become  more  important  in  the  future. 
Cycle  time  is  how  long  it  takes  to  complete  a 
product  or  service  from  the  start  date  to  the 
delivery  date.  Cycle  time  consists  of 
processing  time,  transit  time,  waiting  time, 


review  time,  and  rework  time.  Much  more 
will  be  said  on  this  matter  later  in  the  book. 

Value  Added 

Next  to  cycle  time,  value  added  is  the 
next  most  important  attribute  to  measure  and 
control.  It  is  the  value  added  during  the 
production  process  from  the  day  raw  material 
and  ideas  are  combined  to  intermediate 
processing  to  final  delivery.  In  sum,  it  is  the 
value  of  the  supplies,  labor  knowledge, 
energy,  equipment,  etc.,  that  goes  into 
making  a  product  or  producing  a  service.  It 
is  why  things  cost  what  they  do.  It  is  a 
competitive  factor  in  today’s  cost  conscious 
world. 

Waste  Analysis 

Probably  the  third  most  important 
attribute  to  measure  is  waste.  Waste  is  not 
simply  saving  of  paper  clips  and  recycling 
paper  but  is  far  more  evasive.  Waste  is 
connected  to  both  cycle  time  and  value- 
added.  It  is  the  added  expenses  that  are 
passed  on  to  the  customer  from  non-value 
added  activities  such  as  excessive  transit 
time,  unnecessary  review,  frequent  rework 
activities,  and  uncalled  for  waiting  time. 
Waste  is  said  to  account  for  between  10% 
and  95  %  of  value  added  costs  (depending  on 
the  organization  and  what  is  being  produced) 
with  a  mean  value  typically  around  35%  of 
an  organization’s  budget.  Much  more  will  be 
said  about  this  important  performance 
measure. 
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Flexibility /Adaptability  Analysis 

Organizations  which  do  not  adopt  to  the 
new  paradigms  of  the  current  surge  in  TQM 
and  continuous  improvement  activities  will 
not  exist  by  the  early  part  of  the  21st  century. 
This  factor  is  one  of  he  reasons  why  Japanese 
products  and  services  have  soared  in  the  last 
20  -  30  years.  They  have  adopted  the  new 
tools,  techniques,  and  procedures  to  produce 
high  quality  and  low  cost  products  and 
services.  They  have  passed  through  the 
“cultural  change”  barrier  that  so  haunts  the 
West.  The  enclosed  material  discusses 
measures  of  adaptability  in  some  detail  not 
only  as  a  specific  subject  but  tacitly 
throughout  the  material. 

Benchmarking 

Benchmarking  is  the  comparing  of  your 
performance  with  that  of  a  competitor,  a 
world  class  organization,  the  best  in  a  class, 
or  all  of  these  references.  It  is  a  way  of 
telling  an  organization  where  it  stands 
relative  to  a  reference  point.  It  also  allows 
an  organization  to  set  “stretch  goals”  to  meet 
the  challenge  of  knowing  where  they  stand. 
There  are  a  number  of  approaches  to 
benchmarking  and  they  are  discussed  in  the 
enclosed  material. 

Shewhart  Control  Charts 

The  Shewhart  Control  Chart  is  most 
likely  the  single  most  important  technique  in 
a  TQM,  APIC  toolbox.  It  allows  the  user  to 
identify  the  problems  associated  with 
organizational  quality  performance,  cycle 
time  analysis,  value-added  analysis,  and 


waste-all  in  one  package.  It  is  a  graphical 
tool  and  is  used  to  diagnose  and  aid  in 
locating  performance  problems  in  a  dynamic 
sense.  Most  tools  measure  performance  after 
the  fact;  after  the  product  or  service  is 
already  produced  and  in  the  customer’s 
hands.  The  Shewhart  control  chart  allows 
the  analysts  to  observe  problems  as  they 
occur;  to  detect  and  aid  in  diagnosing  errors 
before  they  get  built  into  the  product  or 
service.  One  of  the  shortcomings  of 
Shewhart  control  charts  is  that  the  standard 
form  allows  observations  of  only  one  process 
at  a  time.  The  standard  form  still  stands  as 
one  of  the  most  powerful  process  analysis 
tools  available  today.  An  index  of 
performance  is  calculated  from  process 
measurement  using  the  Shewhart  control 
chart. 

Demos  Control  Charts 

The  Demos  control  chart  is  an  expansion 
of  the  Shewhart  control  chart  which  allows 
the  analyst  to  trace  back  through  the  system 
or  process  to  other  sub-process  control  charts 
to  “pin  down”  the  exact  source  of  a  problem. 
The  text  provides  an  example  of  this 
technique  using  extensive  details. 

PERT 

The  Program  Evaluation  and  Review 
Technique  (PERT)  was  originally  designed  to 
monitor  the  one-time  scheduling  of  a  large 
scale  project.  It  was  so  successful  that  it 
became  a  tool  which  is  adapatable  to 
monitoring  projects  of  all  sorts.  It  was 
intended  to  serve  as  a  program  guide  or  map 
for  complex  projects  but  is  now  being  used  to 
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model  alternative  organizational  procedures. 
It  is  also  very  useful  in  testing  alternative 
plans  and  employing  the  “what  if”  to  test 
cause  and  effect.  It  is  ideal  for  process 
mapping  as  will  be  demonstrated  in  this  text. 

Benefit-to-Cost  Ratio  Analysis 

The  benefit-to-cost  ratio  technique  is 
familiar  to  the  COE.  It  is  a  technique  that 
requires  that  the  benefit  of  a  proposed  action 
and  its  costs  both  be  developed  to  determine 
if  an  action  or  project  is  economically 
justified.  If  the  benefits  are  divided  by  the 
costs  with  a  result  which  is  greater  than  1.00, 
then  the  proposed  action  is  justified  on 
economic  grounds.  If  the  resulting  ratio  is 
equal  to  1.00,  then  the  proposed  action  is 
break-even.  Finally,  if  the  resulting  ratio  is 
less  than  one,  the  action  is  not  justified. 
Both  the  benefits  and  cost  must  be  in 
capatible  annual  terms.  The  technique  is 
excellent  for  evaluating  the  impacts  and 
justification  for  such  actions  as 
reengineering,  reallignment  studies, 
downsizing  or  related  activites  and  studies. 
Currently,  there  are  very  few  credible  studies 
ongoing  which  have  evaluated  organizational 
changes  in  this  manner.  In  many  cases,  the 
original  rational  was  not  objective. 

Quality  Function  Deployment 

Quality  Function  Deployment  (QFD)  is 
the  “voice  of  the  customer”.  It  is  a  tabular 
technique  for  sytematically  determining 
customer  wants  and  needs,  the  amount  of 
these  wants,  how  they  are  going  to  be 
achieved,  what  is  going  to  be  achieved, 
prioritizing  goals  and  objectives,  and 


benchmarking-all  in  one  technique.  It  is  an 
excellent  customer  satisfaction  tool  easily 
employed  or  adopted  to  almost  any 
production  or  service  operation.  QFD  began 
as  an  engineering  tool  to  assure  that  the 
development  process  led  to  a  product  that 
would  meet  customer  expectations.  It  has 
developed  into  a  holistic  planning  tool.  It  is 
also  a  preventive  tool  designed  in  part  to 
prevent  last  minute  surprises  often  associated 
with  complex  projects.  It  is  highly 
recommended  that  the  COE  adopt  this  tool  in 
its  engineering  and  planning  processes.  The 
enclosed  examples  will  provide  a  good  deal 
of  information  in  an  applied  example. 

Quantification  of  Goals  and  Objectives 

Many  goals  and  objectives  are  developed 
in  such  a  way  that  they  are  subject  to 
subjective  interpretation  in  purpose  and  in 
outcome.  This  often  happens  because  they 
are  developed  in  a  “fuzzy”  way.  Many  goals 
and  objectives  can  be  made  more 
deterministic  by  constructing  them  in  ways 
that  allow  for  objective  quantification  of 
intent  and  outcome  so  that  there  are  no 
misunderstandings.  It  is  stated  at  the  outset 
that  not  all  goals  and  objectives  are 
quantifiable;  many  that  can  be  are  not 
because  the  writer  is  not  accustomed  to 
preparing  them  in  the  proper  format.  The 
approach  described  in  the  text  shows  how  to 
construct  quantifiable  goals  and  objectives. 
Users  are  encouraged  to  use  this  approach 
rather  than  continue  the  traditional  “soft” 
unverifiable  approach. 
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Customer  Surveys 

The  least  expensive  and  yet  very 
powerful  way  to  find  out  what  is  going  on,  or 
to  produce  data  for  a  specified  purpose  is  to 
generate  a  well  developed  survey.  This 
device  satisfies  much  of  the  GPRA  and  APIC 
requirements  for  listening  to  the  voice  of  the 
internal  and  external  customer.  Surveys  are 
also  very  amenable  to  statistical  analysis  and 
can  be  used  to  benchmark  at  a  point  in  time 
against  a  competitor  or  other  agency  or,  over 
time,  to  assess  changes  in  behavior  or 
perceptions.  Well  developed  surveys  are  the 
key.  A  simple  question  is  not  always 
interpreted  the  same  way  by  different  people. 
The  wording  must  be  as  specific  as 
reasonably  possible.  In  order  to  prevent 
bias,  survey  questions  should  be  developed 
by  an  outside  consultant  and  overseen  by 
organization  management  and  employees 
through  their  unions. 

Pareto  Charts 

A  seemingly  trivial  device,  the  Pareto 
chart,  is  extremely  powerful.  It  is  used  to 
rank  problems,  ideas,  and  solutions  for 
taking  action.  It  is  one  of  the  most  used  tools 
by  TQM  organizations  today.  The  last  part 
of  this  book  discusses  the  value  and 
application  of  this  approach  to  problem 
resolution.  It  is  often  used  in  conjunction 
with  focus  groups  or  brainstorming  sessions. 
This  is  where  its  power  is  best  unleashed. 


Applicability  of  Performance 
Measurement  Tools  to  Corps  of  Activities 

Every  performance  measurement  tool 
presented  in  this  text  is  useful  to  COE 
activities.  They  are  not  theoretical  tools  but 
represent  methods  which  have  been  time 
tested  by  actual  application  and  all  are 
currently  being  used  by  “cutting  edge”  and 
“world  class”  organizations.  They  produce 
results!  Some  are  more  applicable  to  certain 
facets  of  an  organaization  than  others. 

Table  1  illustrates  the  usefulness  of  each 
tool  to  major  COE  requirements;  present  and 
future.  Many  of  these  COE  requirements 
overlap  to  a  considerable  degree.  This  is 
apparent  in  the  table.  Rather  than  treat  these 
requirements  as  separate  tasks,  it  is  advised 
that  they  all  be  absorbed  into  the  Baldrige 
(APIC)  criteria  because,  in  fact,  they  belong 
there. 

As  they  currently  exist,  some  of  the 
criteria  have  been  in  “paradigm  paralysis” 
for  some  years  and  need  to  be  re-vamped  to 
conform  with  what  works  to  make  an 
organization  the  best  in  today’s  highly 
competitive  environment.  Whatever  is  done 
along  these  lines.  DO  NOT  MODIFY 
BALDRIGE  (APIQ  TO  FIT  THE  CORPS: 
THE  CORES  NEEDS  TO  BE  CHANGED 
TO  FIT  BALDRIGE  (APIC).  TO  DQ 
OTHERWISE  WOULD  TOTALLY 
defeat  THE  PURPOSE  QE  THE 

CONSTRUCTIVE  CHANGE  CURRENTLY 

TEANSEORMINGTHE  ORGANIZATION 
FOR  THE  21ST  CENTURY.  I_T  WOULD 
BE  A  SIEE  BACKWARD  AND 
INSTnVTWNALLY DESTRUCTIVE  The 
last  three  sentences  above  were  typed  in 
bold,  italics,  and  underlined  for  good  reason! 
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This  is  not  an  area  for  compromise.  Bending 
the  intent  or  using  clever  re-wording  or 
interpretation  of  the  Baldrige  criteria  to 
obtain  “buy-in”  would  start  a  self-inflicted 
virus  leading  to  a  return  of  the  bureaucratic 


disease.  Change  the  Corps  to  meet  Baldrige. 
Do  not  change  Baldrige  to  satisfy  the  Corps. 

The  following  appendices  will  go  on  to 
describe  the  details  of  the  previously 
discussed  performance  measures. 
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APPENDIX  A 


PERFORMANCE  MEASUREMENT 
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A  11  performance  measures  are  faulty 

in  that  they  cannot  take  into  consideration  all 
the  factors  that  contribute  to  what  it  is  they 
are  measuring.  In  this  context  they  are 
essentially  proxies.  There  are  also 
circumstances  where  performance  measures 
provide  contradictory  readings.  When 
measuring  an  attribute  or  characteristic,  one 
measure  may  indicate  an  improvement  while 
another  may  show  matters  deteriorating.  This 
presents  a  good  case  for  using  more  than  two 
measures  of  the  same  thing  and  a  good 
reason  to  use  more  than  one  technique  or 
method.  Using  multiple  measures  of  the 
same  attribute  often  creates  “families  of 
measures”.  The  number  of  ways  to  measure 
performance  abound.  Some  methods  are 
simple  and  effective  and  a  few  are  so 
complex  that  one  wonders  if  they  are 
measuring  anything  at  all.  This  appendix 
will  focus  on  those  which  will  allow  COE 
analysts  to  do  more  with  less.  It  will  also 
envelop  only  those  methods  which  are  COE 
relevant.  The  COE  is  a  government 
institution  and  not  profit  driven,  consequently 
market  price  related  performance  techniques 
will  not  be  covered.  The  reader  may  refer  to 
the  bibliography  for  private  sector  techniques 
and  methods. 

Measurement  techniques  will  cover  the 
areas  of  input  quantification,  productivity, 
effectiveness,  efficiency,  output,  and 
outcome.  All  are  needed  to  satisfy  the  spirit 
of  the  NPR,  GPRA  and  the  APIC  programs. 


Twenty  tools  and  techniques  will  be 
presented  with  reference  to  others  in  the 
literature.  The  tools  and  techniques  are:  uses 
of  raw  data,  partial  and  multifactor 
productivity  indices,  weighted  relative 
aggregates  productivity  (cost  and  production) 
indexes,  the  benefit-to-cost  ratio  (BCR), 
network  schematics  or  mapping,  statistical 
process  control  (SPC),  benchmarking,  and 
customer  surveys-internal  and  external, 
among  others.  Collectively  these  tools  and 
techniques,  which  are  prevention-based,  can 
be  used  to  diagnose  problems,  quantify 
activities, produce  continuous  process  control 
in  a  total  quality  management  environment. 
Also  note  that  the  methods  presented  in  this 
report  are  useful  for  the  performance 
measurement  of  projects,  programs,  financial 
management,  business  practices,  KRA's,  or 
systems  and  processes.  Table  la  presents  a 
comprehensive  view  of  user  utility  for  each 
type  of  performance  measure.  All  are  very 
suitable  for  graphical  displays  but  some  may 
be  more  or  less  appropriate  for  customer 
viewing. 

There  are  three  terms  used  throughout 
the  discussions  on  index  numbers  which  need 
to  be  defined.  These  are  the  terms 
aggregate,  simple,  and  percentage  relative  or 
just  relative.  The  word  aggregate  means  the 
sum  of  a  number  of  terms,  items,  or  elements 
which  make  up  a  total.  The  term  simple 
refers  to  an  unweighted  number.  The  term 
percentage  relative  or  just  relative  is  defined 
as  the  ratio  of  a  current  value  to  a  base  value; 
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this  becomes  a  percentage  relative  when 
multiplied  by  100. 

There  are  also  three  basic  types  of  index 
numbers  including  cost  indicators  (cost  per 
unit),  quantity  indicators,  and  value 
indicators  (cost  per  unit  times  quantity).  The 
reader  should  keep  in  mind  that  one  of  the 
advantages  of  many  of  the  indexes  discussed 
in  this  book  is  that  they  allow  for  the 
inclusion  of  multiple  performance  measures 
to  produce  a  single  index.  These  will  be  used 
and  explained  throughout  the  following 
pages. 

Inflation  And  Deflators 

In  the  various  indexes  that  will  be 
discussed  presently,  the  issue  of  adjusting 
monetary  numbers  such  as  costs  and  value 
needs  attention.  This  adjustment  is  necessary 
to  account  for  the  oscillating  value  of  the 
dollar  over  time.  Correction  involves 
dividing  the  unadjusted  value  by  an 
appropriate  price  level  index  called  a  deflator 
which  compensates  for  the  changing 
purchasing  power  of  money  over  time.  It  is 
necessary  to  make  this  correction  so  that  real 
purchasing  is  being  evaluated  and  not  inflated 
dollars.  This  topic  will  be  covered  in  more 
detail  later  in  the  report  but  needs  to  be 
mentioned  here  to  increase  awareness  of  the 
value  of  the  index  numbers  being  discussed. 

Productivity  and  Efficiency  Defined 

Productivity  and  efficiency  are  both  part 
of  performance  measurement.  If  you  were  to 
ask  different  disciplines  what  the  meaning  of 
each  of  these  terms  are,  you  would  get 
different  answers.  Engineers,  economists, 


politicians,  psychologists,  sociologists,  etc., 
view  the  world  differently.  All  these 
perceptions  are  correct;  it  is  a  point  of  view 
that  makes  the  difference.  Because  of  this, 
this  text  will  not  attempt  to  precisely  define 
each  of  these  terms  but  will  adopt  a  generic 
definition  which  better  suits  the  diverse 
Corps  population  who  will  be  likely  to  use 
this  material.  Even  though  definitions  may 
differ  somewhat,  the  tools  which  measure 
productivity  and  efficiency  are  virtually  the 
same. 

Productivity  "is  a  relationship  between 
quantities  of  outputs  from  a  system  and 
quantities  of  inputs  into  the  same  system " 
[Sink,  1985],  This  is  mathematically 
expressed  as  Output/Input.  The  real  thrust 
behind  productivity  is  to  get  out  of  a  resource 
input  more  than  the  cost  of  the  input. 
Families  of  productivity  and  efficiency 
measures  are  usually  more  useful  than  single 
measures.  They  present  a  mosaic  of  what  is 
happening  as  compared  to  single  measures, 
as  mentioned  earlier.  Productivity  growth  is 
an  increase  in  output  that  exceeds  the  growth 
in  inputs.  Efficiency  "is  the  degree  the 
system  utilized  the  "right"  things  or  "the 
comparison  between  resources  we  expect  or 
intend  to  consume  in  accomplishing  specific 
goals,  objectives,  activities,  and  resources 
actually  consumed"  [Sink,  1985]. 

Before  proceeding  it  must  be  recognized  that 
productivity  increases  occur  for  five 
circumstances: 

1 .  Output  increases  while  input 
decreases. 

2.  Output  increases  while  inputs  remain 
constant. 
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3.  Output  increases  and  input  increases, 
but  at  a  lower  rate. 

4.  Output  remains  constant  while  input 
decreases. 

5.  Output  decreases  and  input 
decreases,  but  at  a  more  rapid  rate. 

With  these  working  definitions  in  mind 
we  can  proceed  with  the  measurement  of 
productivity  and  efficiency;  and  later,  with 
effectiveness. 

Simple  Univariate  (RAW) 
Descriptive  Statistics 

Univariate  data  are  the  easiest  to  use, 
most  direct,  and  most  generally  understood 
by  analyst  and  customer  alike.  Raw  data  are 
simple  unadjusted  numbers.  They  are  the 
numbers  we  use  daily  and  often  have  one 
dimensional  units  of  measure  such  as  feet, 
dollars,  kilowatts,  acres,  species,  visitors, 
etc.  They  are  used  for  graphs,  tables  and 
charts  that  display  data  so  that  they  are  easy 
to  understand.  They  have  little  interpretation 
beyond  simple  description. 

Univariate  values  are  most  useful  when 
plotted  and  observed  over  time  to  expose 
trends  and  patterns  in  data.  They  are  also 
useful  in  benchmarking  (comparing  your 
operation  with  the  best  in  the  business). 
Figure  la  portrays  a  typical  example  of  raw 
data  values  plotted  over  time  and  used  in 
benchmarking.  In  this  example,  District  A  is 
comparing  a  port  located  within  its 
boundaries  with  that  of  a  benchmark  port 
located  in  another  jurisdiction.  Plotting  the 
performance  of  "tons"  over  time  shows  that 
the  growth  in  tonnage  of  Port  A  had 
surpassed  that  of  the  benchmark  port  in 


1991.  Both  ports  are  showing  increasing 
tonnages  over  time  and  both  indicate  a 
leveling  trend  which  started  around  1994. 
The  performance  of  Port  A  has  been  better 
than  that  of  the  benchmark  port  as 
demonstrated  by  a  more  rapid  growth  from 
1985  to  around  1993.  Notice  also  that  a 
rapid  spike  in  growth  occurred  in  Port  A  in 
1990.  This  sort  of  illustration  shows  the 
effectiveness  of  using  simple  data.  They 
communicate  a  good  deal  of  information. 
They  are  useful  for  examining  historic 
trends,  in  forecasting;  they  point  out  unusual 
or  extreme  events,  and  are  good  for 
benchmarking.  It  is  expected  that  they  will  be 
used  frequently  in  performance  evaluation. 
All  of  this  could  be  valuable  information  in 
conducting  studies  of  port  expansion  and  in 
making  a  case  for  funding  studies.  Let  us 
examine  raw  numbers  from  another 
perspective. 

To  state  that  the  COE  protected 
2,000,000  acres  of  wetlands  in  fiscal  year 
1997  may  be  meaningless  in  itself.  To  state 
that  the  number  of  protected  acres  in  fiscal 
year  1996  was  1,500,000  adds  value  to  the 
information;  there  has  been  an  increase  of 
500,000  acres  which  represents  a  gain  of 
33.3  percent  over  a  one  year  period.  The 
continued  record  keeping  and  plotting  of  this 
information  over  time  makes  this  simple 
information  highly  valuable  in  defending 
program  schedules  and  budgets.  It  represents 
essential  feedback  to  the  process,  makes  for 
excellent  public  relations,  and  is  also 
valuable  for  benchmarking  purposes.  These 
numbers  are  powerful  and  useful  but  could 
also  be  very  deceiving. 

Simple  numerical  measures  of 
performance  may  tell  only  part  of  the  story. 
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Low  Software  is  available  whicJh  will  reduce  costs  of  use. 

Medium  Training  is  provided  for  knowledge  &  skill  development  and  for  universal  consistency  i 

High 


They  do  not  tell  how  much  labor  resources 
were  used  as  input,  the  capital  (an  economic 
term  for  equipment)  expenditures  required, 
any  reference  to  quality  characteristics  of  the 
attribute,  the  costs  involved,  the  technology 
employed,  or  the  knowledge  required,  among 
other  factors.  For  example,  the  500,000 
acreage  increase  cited  above  may  have  cost 
twice  as  much  per  acre  to  preserve  as  did  the 
previous  1,500,000  acres.  The 

environmental  characteristics  of  the  acreage 
may  be  marginal  as  compared  to  prime,  high 
quality  wetlands  preserved  in  the  past. 
Under  this  scenario,  the  500,000  acres  may 
not  present  themselves  as  an  impressive 
accomplishment.  It  must  be  emphasized  that 
the  presentation  of  simple  numerical  values 
must  be  carefully  qualified.  If  the  numbers 
are  "clean"  and  developed  under  conditions 
comparable  to  past  efforts,  then  raw  data  are 
a  good  measure.  Otherwise,  buyer  beware. 
Despite  their  importance  simple  facts  may  be 
insufficient  in  providing  a  basis  for  action. 
Effective  action  is  more  a  function  of 
determining  cause  and  effect  connections 
among  processes  and  between  process  and 
results. 

The  Use  Of  Ratios 

The  use  of  ratios  is  a  technique  most 
performance  analysts  are  already  acquainted 
with.  A  ratio  is  simply  one  number  divided 
by  another-the  purpose  is  to  determine  or  to 
create  a  relationship  between  a  numerator 
and  denominator  for  comparative  purposes. 

The  usual  association  of  two  such 
numbers  helps  to  determine  what  part  or 
share  one  number  is  of  another,  to  create  a 


percent  by  multiplying  the  dividend  by  100, 
or  measure  changing  relationships  over  time. 

Ratios  are  functionally  used  to  examine 
and  explore  cross  sectional  data  or  time 
series  data.  A  cross  sectional  relationship  is 
one  which  holds  time  constant  to  show 
comparative  association  at  a  point  in  time. 
Table  2a  provides  an  example  of  this  type  of 
association.  In  this  example  the  ratios  show 
that  a  COE  district's  Xerox  expenses  increase 
with  FTE  levels  which  might  be  expected. 
It  also  shows  that  this  relationship  grows  out 
of  proportion  as  district  sizes  approach  771 
FTE's  or  larger.  This  is  a  cost-of-doing- 
business  issue  which  may  provide  a  clue  to 
organizational  operation  and  economies  of 
scale.  Why  is  this  occurring?  Is  it  due  to 
more  work  or  more  bureaucratic  waste? 
Notice  that  this  analysis  is  for  a  given  year 
only.  No  time  has  passed. 

The  use  of  time  driven  ratios  are  found 
by  dividing  the  raw  data  value  in  succeeding 
years  by  the  raw  data  value  found  in  the  base 
year.  The  general  expression  for  this  ratio  is: 

I  =  (Cn/C0)  x  100  for  cost  analysis 
and 

I  =  (Qn/Q0)  x  100  for  quantity 
analysis 

where: 

I = Index 

Cn  =  Cost  in  year  n, 

C0  =  Cost  in  the  base  year, 

Qn  =  Quantity  in  year  n, 

Q0  =  Quantity  in  the  base  year 

The  data  may  be  either  input  data  or 
output  data.  Table  3a  provides  example  of 
the  use  of  time  ratios.  The  ratios  are  useful 
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TABLE  2a 

CROSS  SECTIONAL  ANALYSIS 

1997  1 

§  a) 

||  District  Size 

(2) 

(3)  1 

(FTE’s) 

Xerox*  Expenses 

Ratio  (2)/(l) 

||  350 

$100,000 

$286/FTE  || 

|l  410 

$125,000 

S305/FTE  || 

II  528 

$170,000 

S322/FTE  || 

|  600 

$210,000 

S350/FTK  || 

SI  771 

$385,000 

S499/FTE  || 

1  903 

$500,000 

$554/FTE  || 

*Include  purchases,  maintenance,  paper,  etc. 

for  showing  the  change  in  the  value  of  some 
factor  being  measured  over  time  relative  to  a 
base  or  starting  year  value.  They  should  not 
be  used  without  elaboration  as  they  can  be 
misleading  for  reasons  cited  earlier  in  the 
discussion  of  raw  data  usage.  If  costs,  price, 
values,  or  other  dollar  measures  are  used, 
care  must  be  taken  to  adjust  for  inflation 
using  price  deflators.  As  the  table  illustrates, 
ratios  can  be  developed  based  upon  given 
year  to  base  year  values  or  from 
accumulative  data  to  a  base  year  value. 

Special  Purpose  Indexes 

There  are  circumstances  where  the 
previous  performance  indexes  are  not 


applicable  and  another  approach  is  needed. 
Special  purpose  indexes  are  designed  to  help 
fill  this  void.  Like  some  of  the  previous 
indexes,  special  purpose  indexes  require 
weighting  of  values  by  using  some  criteria 
which  is  indicative  of  the  importance  of  the 
numbers  involved.  This  type  of  index  also 
has  the  useful  characteristic  of  being 
invariant  to  the  units  of  measure  being  used. 
The  procedure  also  does  well  in  allowing  the 
use  of  multiple  variables  to  form  one  measure 
of  performance.  Table  4a  presents  an 
example  application  of  the  use  of  a  special 
purpose  index. 

The  example  uses  the  KRA  called  Work 
Life  Quality  (WLQ).  Work  Life  Quality  is 
concerned  with  issues  related  to  employee 
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TABLE  3a  -  EXAMPLES  OF  TIME  RATIOS 


Year-to-Year  Examples 


Example  1 

if  Year 

Deep  Draft 
Vessel  Delays 

Ratio 

||  1990 

5500  vessel  hrs 

1.00 

l|  1991 

5700 

1.04 

||  1992 

6000 

1.09 

||  1993 

5900 

1.07 

||  1994 

6200 

1.13 

||  1995 

6000 

1.09 

||  1996 

7100 

1.29 

Example  2 


Flood  Damage 
Prevented 


1,200,000 


1,400,000 


1,300,000 


1,200,000 


1,500,000 


Cumulation  Examples 


Example  4 


Cumulative  Kwhrs 
Generated 


w&m 

rir  Mi 

yy 
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safety,  training,  morale  and  EEO/AA 
progression.  Suppose  that  four  attributes 
were  selected  by  management  and  employees 
to  represent  WLQ  proxies.  They  include: 
self  directed  work  teams,  hours  of  training, 
employee  ideas  adopted,  and  the  dollar 
amount  of  incentive  awards  given.  These 
attributes  are  known  to  increase  motivation, 
productivity,  efficiency,  and  effectiveness. 
Consequently  both  management  and 
employees  gain  by  their  use  in  measuring  the 
WLQ.  Each  of  these  attributes  needs  a  little 
explanation  before  proceeding  with  the  point 
at  hand. 

Self  directed  teams  go  beyond  normal 
teamwork  practices  in  that  employees  make 
all  the  decisions  and  do  all  the  planning  for 
the  projects  to  which  they  are  assigned. 
Management  serves  to  assist  employees  only 
at  the  request  of  the  teams  and  then  only  in 
an  advisory,  teacher,  and  coach  capacity. 
Employees  are  fully  empowered.  This  is  a 
highly  desirable  working  environment  for 
professional  employees  and  in  numerous 
studies,  has  proven  to  yield  the  greatest 
productivity  and  efficiency  of  all  working 
relationships  [Manz  and  Sims,  1993].  Self 
directed  teams  avoid  the  autocratic  and 
bureaucratic  nature  of  the  traditional  approach 
to  organizational  operations. 

Training  is  another  important  WLQ 
characteristic  and  is  closely  interwoven  with 
self  directed  work  teams.  Multi-disciplinary 
and  multi-skilled  employees  are  paramount  to 
the  success  of  the  self-directed  team.  It  is  also 
known  that  training  is  a  positive  motivator 
for  employees.  Most  seek  training  for 
increased  development  and  personal  growth. 
In  addition,  studies  have  shown  that  the 


training  of  employees  brings  dividends  to  an 
organization  in  the  long  run. 

The  adoption  of  employee  ideas  is 
another  known  motivating  factor.  When 
employees  know  that  management  and 
leadership  take  their  ideas  seriously,  they 
respond  in  a  positive  fashion.  This  goes 
hand-in-hand  with  incentive  awards. 
Incentive  awards  need  not  be  cash  but  may 
take  a  variety  of  forms  including  time  off, 
honorary  recognition,  prestige,  and  increased 
responsibility  and  opportunity  on  the  job. 

The  four  attributes  used  in  this  example 
are  not  important  to  the  technique  per  se. 
The  essential  idea  is  that  there  has  been  a 
consensus  agreement  between  employees  and 
management  that  these  attributes  are  mutually 
beneficial  and,  more  importantly,  each 
attribute  has  been  given  an  importance 
weight  by  mutual  agreement  as  well.  The 
number  of  employees  belonging  to  working 
self  directed  teams  has  been  given  a  weight 
of  40  percent,  hours  of  training  per  year  is 
given  a  weight  of  30  percent,  and  the 
adoption  of  good  employee  ideas  and 
incentive  awards  have  been  weighed  20  and 
10  percent,  respectively. 

Column  5  of  Table  4a  represents  the 
desired  index  measure  used  to  gauge  WLQ 
performance-an  important  KRA.  It  is 
evaluated  by  dividing  given  year  values  by 
base  year  values,  multiplying  the  results  of 
this  by  the  weighting  factor,  and  then 
summed  across  a  row.  As  the  value  of  the 
index  rises  over  time,  so  does  the  WLQ.  In 
this  example,  the  index  has  risen  from  a  base 
year  (1996)  value  of  100.0  to  a  1998  value  of 
280.0.  From  1998  to  1999,  the  performance 
index  has  declined  to  87  percent  indicating 
that  something  has  gone  awry.  The  index  is 
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sensitive  to  the  change  in  weights  and  the 
number  of  attributes  comprising  the  index. 

This  approach  to  index  number 
development  is  very  flexible  but  carries  with 
it  a  risk  of  bias.  The  weights  are  susceptible 
to  bias  if  not  developed  in  a  scientific, 
supportable  or  verifiable  manner.  The 
example  in  Table  4a  was  selected  for  a 
particular  reason.  Judgement  can  play  a  role 
in  the  development  of  the  weights  but  the 
process  of  weight  selection  needs  to  evolve 
from  consensus.  Only  then  will  the 
performance  index  carry  with  it  real  authority 
and  support.  Traditional  weighting  also 
plays  an  important  role.  Customer  surveys, 
variable  size,  costs,  demand,  etc.,  can  also 
be  used  as  weights.  These  tend  to  be  more 
scientific  because  their  basis  is  in  fact  rather 
than  judgement. 

The  special  purpose  index,  like  all  the 
other  indexes  in  this  text,  can  be  used  to 
measure  performance  in  KRA's,  projects, 
programs,  organizational  operations  and 
financial  well-being.  Weights  can  be 
developed  by  surveys,  budget  proportions, 
product  demand,  etc.,  depending  upon  its 
intended  purpose.  It  should  be  used  with 
care  because  the  potential  for  subjectivity 
creeping  in  is  all  too  real. 

General  Notes  On  Partial  And 
Multifactor  Indexes 

Partial  and  multifactor  indices  convey 
more  information  than  simple  numerical 
values  as  measures  of  performance.  A 
partial  productivity  measure  looks  at  output 
as  a  function  of  a  single  input.  Multifactor 
indexes  look  at  output  as  a  function  of  labor, 
capital  and  other  resource  inputs.  If  only  one 


resource  input  is  used,  the  ratio  is  called  a 
partial  productivity  measure.  If  more  than 
one  resource  input  is  used,  the  ratio  formed 
is  called  a  multifactor  productivity  measure. 
If  all  resource  inputs  are  used,  the  final  ratio 
is  called  a  total  factor  productivity  measure. 
The  distinction  between  the  multifactor  and 
total  factor  productivity  measures  is  basically 
academic  in  terms  of  the  resource  input  that 
are  used  by  the  COE.  The  single  most 
significant  resource  input  for  the  COE  is 
professional  labor. 

Each  can  be  used  to  spot  check  the 
performance  at  a  point  in  an  operation  or  to 
examine  the  final  output  of  a  total  process. 
The  output  of  process  (1)  and  of  process  (4) 
in  Figure  3  (see  Systems  and  Processes)  are 
good  locations  to  conduct  a  spot  process 
check.  The  output  of  processes  (2)  and  (3)  in 
Figure  3  are  a  good  location  to  check  the 
performance  of  the  system.  Partial  and 
multifactor  indexes  are  frequently  quoted  and 
used  in  the  private  sector.  These  indexes  are 
often  constructed  as  the  ratio  of  output  to 
input  and  are  useful  when  benchmarking. 
They  are  also  typically  found  in  measuring 
organizational  productivity.  Mathematically 
the  basic  form  of  this  index  is  [Stevenson, 
1993]: 

Productivity  =  Output/Input 

Like  simple  numbers,  partial 
productivity  indexes  are  made  more  useful  if 
plotted  over  time  and  used  in  benchmarking. 
One  popular  partial  index  is  labor 
productivity  and  the  second  is  capital 
productivity.  Labor  productivity  is  the  most 
prevalent  of  all  partial  productivity 
measurements.  It  is  generally  defined  as  the 
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total  value  of  products  and  services  in 
deflated  dollar  terms  of  products  per  hour  of 
labor  input.  This  is  due  to  the  fact  that  labor 
input  is  involved  in  all  facets  of  production 
and  has  been  the  most  easily  measured 
[Jerome,  1986].  It  reflects  output  relative  to 
labor  hours  worked  or  the  number  of 
employees  used.  Capital  productivity,  on  the 
other  hand,  is  the  value  of  products  and 
services  produced  in  constant  dollars  of 
purchasing  power  per  unit  of  capital  services 
input.  Capital  productivity  is  a  very  useful 
productivity  tool  but  as  will  be  seen  later,  it 
is  very  difficult  to  develop. 

An  example  of  the  construction  of  a 
partial  labor  productivity  index  would 
include: 

□  Number  of  engineer  drawings/Man- 
years  of  labor  =  Number  of 
drawings  per  man-year 

□  Number  of  design  errors 
made/Number  of  employees  = 
Number  of  design  errors  per 
employee 

The  issues  are  more  involved  than  this 
but  the  examples  do  illustrate  the  general 
concepts  and  principles  involved.  This  topic 
will  be  more  fully  developed  in  the 
paragraphs  below. 

Another  useful  characteristic  of  partial 
productivity  ratios  is  their  value  to  making 
forecasts.  For  example,  if  non-contract 
output  to  employee  ratio  is  $140,000  in 
output  per  employee  then  if  the  number  of 
employees  goes  from  475  down  to  450  the 
change  in  expenditure  capability  is  downward 


by:  [$140,000(474-450)]  =  $3,360,000. 
More  contracting  would  have  to  be  done  as 
more  funding  is  to  be  executed.  This  is  a 
good  way  to  determine  future  planning  needs 
required  to  off-set  changes  in  FTE  levels. 
Other  ratios  can  be  developed  to  assist  in 
planning  future  organizational  needs  as  well. 
These  will  become  more  apparent  as  the  text 
continues. 

Partial  Productivity  Measures 

In  measuring  productivity,  the  value  or 
quantity  of  outputs  (products  and  services)  is 
divided  by  resource  inputs  costs.  Partial 
productivity  measurement  employs  the  total 
value  or  quantity  of  output  divided  by  the 
total  value  or  quantity  of  labor  or  some  other 
single  factor  used.  This  approach  makes 
assumptions  about  the  single  factor  input-that 
it  is  the  driving  force  of  production  activities. 
Care  must  be  taken  to  describe  the 
production  process.  For  example,  two 
employees  using  high  technology  equipment 
(computers)  will  be  more  productive  than  ten 
employees  using  hand  calculators  given  the 
same  tasks.  This  situation  makes  comparison 
of  output  per  employee  between 
organizations  misleading.  An  uninformed 
observer  may  be  misled  in  believing  that  the 
ten  employees  are  unproductive  or  even  lazy 
as  compared  to  the  two  employee 
organization!  The  same  misunderstanding 
may  occur  if  the  output  per  dollar  of  capital 
is  used  without  qualification.  If  the 
assumption  is  made  that  two  organizations 
have  roughly  equivalent  capital  resources, 
training,  education,  man-years  of  experience, 
etc.  then  output  per  employee  comparisons 
are  good  benchmarks.  Alternatively,  these 
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may  be  cues  for  adopting  benchmark 
practices.  The  index  may  be  saying  that  your 
organization's  training,  experience  base, 
capital  resources  and  so-on  are  not  up  to  pare 
with  those  of  the  benchmark  organization. 
This  same  assumption  must  be  extended  into 
the  future  if  the  same  comparison  partners 
are  going  to  be  used. 

Table  5a  provides  an  illustration  of  how 
partial  productivity  indexes  are  developed 
with  a  one  output  scenario  using  labor  as  the 
principle  input.  The  example  shows  that 
thenumber  of  permit  reports  completed  by  an 
Operations  and  Maintenance  Division  has 
been  increasing  from  50  reports  in  1997  to 
65  reports  by  1999.  Labor  inputs  have  also 
increased,  moving  from  400  employee-hours 
of  work  in  1 997  to  420  employee-hours  effort 
by  1999.  The  question  to  be  answered  is,  is 
labor  productivity  increasing,  decreasing,  or 
static?  This  is  an  important  concern  because 
it  is  a  measure  of  organizational  productivity. 
Table  6a  addresses  the  question.  By  dividing 
the  number  of  reports  produced  in  each  out 
year  by  the  quantity  of  reports  produced  in 
the  base  year  (1997)  and  multiplying  the 
results  by  100,  a  percentage  is  formed 
producing  the  Output  Indexes  in  column  2  of 
Table  6a.  This  ratio  increases  from  100.0  in 
1997  to  130.0  by  1999.  It  states  that  output 
is  increasing  over  time  relative  to  the  base 
year.  Similarly,  if  the  employee-hours  for 
each  out  year  in  Table  5a  are  each  divided  by 
the  employee-hours  in  the  base  year  and  the 
result  multiplied  by  100,  another  percentage 
is  formed.  This  is  the  Input  Index  as  shown 
in  column  3  of  Table  6a. 

These  values  grow  from  100.0  in  1997 
to  105.0  in  1999.  Labor  input  is  also 
growing  over  time.  If  the  output  index  is 


divided  by  the  input  index  for  each  year,  a 
partial  productivity  index  is  formed  as  shown 
in  the  last  column  of  Table  5a.  This  index 
progresses  from  100.0  in  1997,  to  117.1  in 
1998,  and  to  123.8  by  1999.  Productivity  has 
increased  rather  dramatically.  The  task  of 
multiplying  each  ratio  by  100  is  a  convention 
which  changes  a  decimal  to  a  percentage  and 
adds  no  other  real  value  to  the  results.  It  is 
easy  to  see  why  the  use  of  partial 
productivity  measures  is  so  popular. 

Table  7a  provides  an  example  of 
productivity  evaluation  using  two  outputs 
instead  of  one  as  was  the  case  in  the  partial 
productivity  example  developed  above.  More 
than  two  are  also  possible.  It  is  still  a  partial 
productivity  measure  because  the  two  factor 
outputs  are  two  of  the  same  resources, 
summed  and  divided  by  a  single  factor  input. 
The  example  consists  of  two  separate 
navigation  channels  in  the  same  district 
undergoing  periodic  dredging  by  a  Hopper 
dredge  as  part  of  an  O&M  operation.  This 
task  requires  that  a  60  foot  channel  depth  be 
maintained  at  all  times  to  facilitate  vessel 
traffic  and  cargo  deliveries.  The  year  1997  is 
the  base  year  being  used  for  time 
comparison.  The  raw  data  show  that  the 
output  from  the  dredging  operations  results 
in  the  removal  of  5,700,000  cubic  yards  of 
material  from  channels  A  and  B  in  1997 
using  a  total  of  281.1  employee-years  of 
work.  The  same  operation  continues  over 
time  requiring  the  removal  of  5,900,000 
cubic  yards  of  material  in  1998  using  245.3 
employee-years  of  labor,  and  the  removal  of 
5,200,000  cubic  yards  in  19992  using  201.1 
employee-years  of  effort.  Both  the  amount 
of  material  removed  and  the  labor  resource 
input  fluctuate  over  time.  The  question  is,  is 
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TABLE  5a 

PARTIAL  PRODUCTIVITY  INDEX 


1  (1) 

SI  Year 

(2) 

Output  (Reports) 

(3)  1 

Input  (Labor  Hours) 

||  1997 

50 

400 

j|  1998 

60 

410  I 

||  1999 

65 

420  || 

TABLE  6a  8 

PARTIAL  PRODUCTIVITY  INDEX  | 

II  (1) 

(2) 

(3) 

(4)  |I 

Year 

Output  Index 

Input  Index 

Partial  Productivity  1 1 

||  1997 

o 

o 

o 

i—H 

100.0 

100.0  1! 

!  1998 

120.0  2 

102.5 

117.1  3  | 

||  1999 

130.0 

105.0 

123.8  |j 

1  Base  Year:  (50/50)  x  100  =  100.0  2  1998:  (60/50)  x  100  =  120  3  (120.0/102.5)  x  100  =  171.1 


this  operation  becoming  more  productive, 
less  productive,  or  unchanged  over  the  three 
year  period? 

Table  8a  provides  an  examination  of  the 
above  question  by  first  weighing  output  unit 
labor  requirements  for  the  base  year.  This 
sets  the  standard  or  basis  for  evaluating  labor 
input  for  the  remaining  two  out  years.  For 
Channel  A,  the  136.0  employee-years  of 
effort  is  divided  by  the  3,000,000  cubic  yards 


of  material  removed.  For  Channel  B,  the 
145.1  man-years  of  effort  is  divided  by  the 
2,700,000  cubic  yards  of  material  excavated. 
Both  quotients  are  expressions  of  inputs  per 
unit  of  output. 

Table  9a  carries  the  analysis  one  step 
further  by  multiplying  the  base  year  weighted 
output  labor  requirements  previously 
calculated  by  output  dredging  quantities  for 
each  channel  and  year.  The  results  are  a 
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TABLE  7a 

O  &  M  Dredge  Material  Removal  -  60  Foot  Channel 
(Program  Performance) 

Basic  Data  for  Two  Outputs 


1997 

(Base  Year) 


Channel  A 


Channel  B 


Total 


1998 


Channel  A 


Channel  B 


Total 


1999 


Channel  A 


Channel  B 


Total 


(Output) 

(Input)  || 

Material  Removed 
(cy) 

Employee- Years  of  Effort  || 

3,000,000 

136.0  I 

2.700.000 

145.1  II 

5,700,000 

281.1  | 

3,200,000 


2.700.000 


5,900,000 


3,500,000 


2.700.000 


5,200,000 


115.2 


130.1 


245.3 


90.2 


110.9 


201.1 
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TABLE  8a 

WEIGHTED  OUTPUT  UNIT  LABOR  REQUIREMENTS 


Output  1997 

Input  1997 

Unit  Labor  Effort 

!!  Channel  A 

3,000,000 

136.0 

136.0/3,000,000  =  0.000453  | 

Channel  B 

2,700,000 

145.1 

145.1/2,700,000  -  0.000537  |j 

weighted  output  with  a  basis  anchored  in 
1997.  When  the  total  weighted  output  for 
each  out  year  are  divided  by  the  base  year 
weighted  output,  a  partial  table  output 
productivity  is  created.  This  number  by 
itself  is  a  valuable  index.  In  this  case,  it 
shows  an  upward  (1998)  trend  followed  by  a 
downward  (1999)  trend  in  dredge  material 
excavation  relative  to  the  base  year.  These 
data  suggest  that,  over  the  three  year  period, 
there  has  been  less  shoaling  and  sediment 
build-up  in  the  channels  thus  requiring  less 
dredge  material  removal  to  maintain  channel 
depths.  As  caveated  earlier,  a  single  index 
may  not  be  telling  the  entire  story.  Table 
10a,  described  below,  adds  additional 
valuable  information. 

Table  10a  illustrates  how  the  input 
indexes  for  each  year  are  evaluated.  The 
total  employee-year  inputs  for  each  year  are 
divided  by  the  base  period,  employee-year 
inputs  to  form  a  productivity  index  of 
inputs.  Notice  that  input  productivity  is  also 
trending  downward  over  time.  This  too,  is 
valuable  information  but  suffers  the  same 
incompleteness  problem  as  all  ratio  indexes. 


One  additional  step  is  needed  to  form  the  two 
output  per  unit  labor  series  index. 

Table  11a  shows  that,  if  each  of  the 
output  indexes  developed  in  Table  9a  are 
divided  by  each  of  the  input  indexes  in  Table 
10a,  a  ratio  of  output  per  unit  labor  input  is 
formed.  This  is  the  two  output  partial 
productivity  sought.  It  says  that  over  the 
1997-1999  period,  the  dredge  removal 
operations  have  become  more  productive. 
The  amount  of  output  per  unit  labor  input  has 
increased.  Stated  another  way,  it  is  costing 
less  to  remove  a  cubic  yard  of  material  in 
1999  than  it  did  in  1997.  This  may  be  due  to 
improved  strategic,  tactical,  or  operational 
planning,  better  process  management 
techniques,  more  experienced  labor,  or  better 
technology.  It  may  also  be  a  combination  of 
a  number  of  factors.  It  may  stem  from  the 
establishment  of  more  definitive  goals  and 
objectives  as  well.  Hopefully,  it  was  due  to 
proactive  management  endeavor  rather  than 
happenstance.  The  reader  can  now  imagine 
how  this  tool  can  be  employed  to  measure 
changes  in  productivity  and  to  test  alternative 
management  practices  against  existing 
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TABLE  10a 

DETERMINATION  OF  INPUT  INDEXES 


Year 


1997 


1998 


1999 


Input  Employee-Years 


281.1 


245.3 


200.1 


Productivity  Index 


100.0 


87.3  1 


71.2  2 


(245.3/281.1)  x  100  =  87.3  2  (200.1/281.1)  x  100  =  71.2 


TABLE  11a 

PRODUCTIVITY  INDEXES 


Year 

Output  Index 

Input  Index 

Partial  Productivity  Output 
per  Employ ee-Year  Input 

1997 

100.0 

100.0 

100.0 

1998 

103.2 

87.3 

118.2  1 

1999 

91.9 

71.2 

129.1  2 

1  (103.2/87.3)  x  100=  118.2  2  (91.9/71.2)  x  100=  129.1 


practices  to  determine  whether  a  newly 
implemented  organizational  structure, 
operational  process,  or  new  goals  and 
objectives  are  having  the  desired  effect.  This 
fits  the  mandates  of  GPRA,  NPR  and  APIC 
quite  satisfactorily.  Remember  also  that  this 
technique  is  useful  in  measuring  the 
productivity  changes  of  a  project,  program,  or 
business  operation,  KRA  performance,  or 
system  and  process  performance. 


Multifactor  Productivity  Indexes 

Simply  defined,  multifactor  productivity 
is  the  value  of  products  and  services,  at 
constant  prices,  produced  per  combined  use 
of  labor,  capital,  and  other  inputs  [Sink, 
1983],  [Thor,  1994],  [Christopher,  1993], 
Multifactor  productivity  is  used  to  help 
overcome  the  shortfalls  in  the  use  of  partial 
productivity  measures.  Partial  productivity 
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measures  such  as  labor,  capital,  energy,  etc. 
fail  to  capture  the  role  of  multiple  resource 
use  as  occurs  when  capital  and  labor  are 
substituted  for  one-another  or  when  their 
combined  effect  is  greater  than  their  simple 
sum.  Comparing  total  output  to  one  input 
overstates  the  importance  of  that  input  when 
all  inputs  have  contributed  to  the  output.  If 
one  or  two  inputs  dominate  the  output 
process  this  situation  is  not  critical,  but,  if 
multiple  inputs  are  used,  then  a  distortion  in 
interpretation  can  result.  Multifactor 
productivity  reflects  the  joint  effects  of  a  mix 
of  labor,  capital,  and  other  resources  used  to 
obtain  a  given  output.  These  include  the 
effects  of  not  only  labor  and  capital,  but 
implicitly  technology,  management  skills, 
economies  of  scale,  knowledge, 
amongothers,  as  well.  Multifactor 
productivity  takes  the  general  form: 

Productivity  =  Total  Output/[Capital  + 
Labor  +  Technology  +  Materials  +  . N) 

where  capital,  labor  technology,  materials, 
and  N  are  any  resource  inputs  which  the 
analyst  wishes  to  include.  Multifactor 
performance  measurement  has  its  basis  in 
market  oriented  institutional  analysis  but  can 
be  adopted  to  fit  the  exploration  of 
government  "firms"  as  well. 

Multifactor  measures  show  changes  in 
many  factors  of  production  per  unit  of  output 
over  a  span  of  time.  They  are  more  complex 
to  adopt  and  often  require  financial  and 
economic  adjustments  before  they  are  used. 
The  most  common  factor  in  the  denominator 
next  to  labor  (and  often  included  with  labor) 
is  capital. 


Using  capital  is  one  of  the  more 
popular  and  most  difficult  variables  to 
incorporate  in  productivity  measurement. 
Capital  is  the  stock  value  of  buildings, 
equipment,  inventory,  and  the  value  of  land. 
The  equipment  aspect  of  capital  often 
explicitly  includes  technology;  this  is  most 
apparent  in  computer  technology,  FAX 
machines,  Xerox  equipment,  etc.  The 
communications  "super  highway"  is,  in  part 
capital,  and  is  one  of  the  more  significant 
reasons  capital  can  add  dramatically  to 
productivity.  All  of  these  go  into  the 
production  of  products  and  services  as  does 
labor.  Notice  that,  in  order  to  evaluate  this 
ratio,  all  values  must  be  in  the  same  units.  In 
order  to  use  capital  to  measure  performance 
it  must  be  converted  into  a  service  measure 
or  flow  concept.  Labor  (the  verb),  on  the 
other  hand,  is  already  a  service  flow.  This 
conversion  can  become  an  intractable 
problem  involving  a  considerable  degree  of 
accounting  experience  and  depreciation 
analysis.  In  order  to  use  capital  as  a  resource 
input  cost  along  with  labor,  it  must  be 
converted  from  a  physical  asset  into  an 
annual  rental  price.  This  has  the  effect  of 
transforming  capital  into  a  flow  concept 
because  it  converts  stock  values  into  annual 
values.  For  example,  what  does  a  $2,000,000 
building  (stock  value)  rent  for  on  an  annual 
basis  (flow  concept),  is  a  stock-to-flow 
concept.  By  definition,  rental  income  from 
constructing  such  a  building  must  cover  the 
cost  of  the  owner  to  pay  off  the  construction 
debt  (mortgage).  Hence  annual  rent  collected 
converts  stock  value  (building  construction  or 
purchase  price)  into  an  annual  payment.  The 
analyst  must  search  the  market  for  the  rental 
cost  of  the  equipment,  buildings,  inventory. 
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and  land  stock.  Once  converted  into  a  flow, 
the  process  allows  all  capital  to  be 
aggregated  into  one  capital  input.  Rental 
prices  are  termed  in  rates  per  constant  dollar 
of  capital  stock.  Each  rental  price  is  adjusted 
for  constant  dollar  capital  stock  to  obtain 
current  dollar  capital  cost.  These  are  then 
converted  into  value  shares  by  various 
methods.  The  process  is  elaborate. 
Multifactor  indexes,  however,  remain 
extremely  valuable  indexes  to  develop  and 
use  for  measuring  performance. 

There  is  another  simpler  way  to  estimate 
capital  flows  which  will  allow  it  to  be 
expressed  in  annual  terms,  added  to  labor 
input,  and  used  together  to  produce  a 
multifactor  productivity  ratio.  The  method  is 
easier  (but  a  little  less  accurate)  than  the 
rental  price  approach.  In  order  to  utilize 
multifactor  analysis  as  a  performance  tool,  an 
example  of  how  to  construct  this  very  useful 
measure  will  be  covered  in  the  following 
paragraphs. 

Table  12a  illustrates  an  example  of  what 
information  is  needed  and  how  it  is  employed 
to  generate  multifactor  indexes.  The  example 
portrays  the  district  level  of  organization  and 
is  close  to  reality  as  data  were  taken  from 
several  district  budget  packages,  rounded  and 
averaged.  The  table  consists  of  four  columns 
showing  the  resource  information  need,  the 
year  1997,  the  year  1998,  and  the 
productivity  indexes  generated,  respectively. 
There  are  also  twenty  rows  showing  how  the 
information  is  used.  Some  of  the  rows  are 
self  explanatory  but  others  will  need 
elaboration. 

Row  1  shows  output  or  what  a  district 
might  call  its  annual  budget.  It  is  the  amount 
of  funds  received  to  carry  out  its  mission. 


Columns  2  and  3  show  monies  that  are  not 
part  of  the  district's  innate  productivity  but 
represent  the  work  activities  of  others. 
Contract  construction  funds  are  mostly 
absorbed  by  construction  activities-not  Corps 
activities.  Similarly,  consultant  contracts  are 
work  done  by  others  that  fill-in  or  enhance 
district  work.  Again,  this  does  not  represent 
funds  directly  used  by  a  district— someone 
else  is  doing  the  work.  The  point  to  be  made 
is  that  these  portions  of  the  budget  must  be 
deducted  from  total  appropriations  because 
they  do  not  directly  reflect  work  done  by 
district  employees,  management  or 
supervision.  Since  contract  dollars  do  not 
reflect  COE  productivity  and  are  removed 
from  analysis  for  this  reason,  a  cost 
adjustment  must  also  be  made  for  contract 
support  expenses  such  as  contract 
administration,  monitoring,  processing,  etc. 
This  is  not  shown  as  a  separate  line  item  but 
is  included  in  items  2  &  3  of  the  table.  We 
do  not  want  to  mix  the  productivity  of  work 
being  done  by  others  with  that  being  done 
directly  by  the  indigenous  work  force.  This 
would  distort  the  results  by  mixing  our 
productivity  with  that  of  others.  Thus  the 
sum  of  these  two  exogenous  funds  need  to  be 
deducted  from  total  appropriations  as  shown 
in  row  4-adjusted  output. 

Prior  to  going  further,  a  new  concept 
needs  to  be  introduced  called  "waste".  As 
described  earlier,  “waste”  consists  of  transit 
time,  time,  waiting  time,  review  time,  and 
rework  time.  These  were  mentioned  earlier. 
These  four  factors  can  actually  be  measured 
by  keeping  track  of  how  much  time  each 
employee  spends  on  each  and  summed,  say, 
by  project.  These  topics  will  be  covered 
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||  TABLE  12a  j 

EVALUATION  OF  PARTIAL  AND  MULTIFACTOR  PRODUCTIVITY  RATIOS 
(Assume  Dollar  Value  Adjusted  for  Inflation) 

1  (1) 

Resource 

(2) 

1997 

(Base  Year) 

(3) 

1998 

(4)  || 

Partial  Productivity  & 

Performance  Ratio  Col.  1 

(3)/(2)  :J 

|  |  1 .  Gross  Output  (Budget)  ($) 

150,000,000 

130,000,000 

0.87  || 

1 1  2.  Construction  Expenditure  ($) 

-82,000,000 

-60,000,000 

0.73  jf| 

1  !j!  3 .  Contracts  for  Labor  ($) 

-5,000,000 

-8,000,000 

1.60  || 

||  4.  Adj.  Output  (1-2-3)  ($) 

63,000,000 

62,000,000 

0.98  : ! 

||  5.  Input,  Employees  (or  FTE’s) 

400 

350 

0.88  || 

||  6.  Avg.  Cost  Per  Employee  1  ($) 

105,000 

108,000 

1.03  || 

P  7.  Total  Payroll  Value  (5  X  6)($) 

42,000,000 

37,800,000 

0.90  || 

||  7a.  Payroll  Adj  for  Waste 

1 1  (7  -  $  waste) 

35,000,000 

(-7,000,000) 

35,800,000 

(-2,000,000) 

1.02  || 

1 1  8.  Input,  Hours  Per  Yr  (FTE  X  2080) 

832,000 

728,000 

0.88  || 

||  9.  Input,  Capital 2  ($) 

18,418,000 

19,300,000 

1.05  || 

1 1  10.  Depreciation  Rate 

6% 

6% 

-  11 

ijl  jij  11.  Depreciation  Expenses  (6%  X  9) 

11  ($) 

1,105,000 

1,158,000 

1.05  || 

||  12.  Long  Term  Return  on 

1 1  Market  Investments 

10% 

10% 

-  11 

||  13.  Inputed  Returns  (10%  X  9)  ($) 

1,842,000 

1,930,000 

1.05  || 

||  14.  Annual  Capital  Input  (11  +  1  3)($) 

2,947,000 

3,088,000 

1.05  || 

||  14a. Capital  Adj.  For  Waste 
|;ji  (14-$  waste) 

2,247,000 

(-500,000) 

2,988,000 

(-100,000) 

L33  || 

||  15.  Output  Per  Employee  (7a/5) 

87,500 

102,286 

L17  \ 

|i;i  16.  Capital/Labor  Ratio 
||  (9/5)  or  (14a/5) 

46,045 

55,143 

1.20  || 

||  17.  Output  Per  Hour  (4/8) 

75.72 

85.17 

(i.i3)  if 

||  18.  Output  Per  Sq.  Ft. 

||  Bldg.  (95,000  s.f.)  18. 

663.16 

652.63 

(0.98)  || 

||  19.  Output/Capital  Ratio  (4/14) 

21,38 

20.08 

0.59  || 

||  20.  Multifactor  Productivity 

1|  4/ (7a  +  14a)/4 

0.59 

0.63 

1.07  || 

1  Includes  salaries  and  all  other  benefits  2  Includes  buildings  and  other  fixed  and  movable  assets. 
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in  more  detail  later  in  the  text  but  a  brief 
description  is  needed  at  this  point.  First, 
waste  is  described  from  the  customer's 
vantage.  It  is  anything  which  prevents  them 
from  getting  the  desired  product  and  service 
at  the  least  cost,  on  time,  and  of  expected  or 
better  quality. 

While  transit  time  in  the  COE  is  a 
relatively  minor  delay  factor  it  nonetheless 
takes  time  away  from  the  customer  receiving 
the  product  or  service  desired. 

Waiting  time  is  a  major  delay  factor  in 
the  COE  process.  It  is  the  time  projects 
spend  in  the  queue  waiting  to  be  processed. 
It  is  not  uncommon  for  a  study  to  wait 
months  for  data,  decisions,  priority,  or 
higher  level  guidance. 

Review  time  is  another  significant  delay 
factor  in  work  completion.  It  is  the  time 
projects  and  studies  spend  undergoing 
internal  and  external  review.  Review  often 
results  in  lengthy  "discussions"  about 
methodology,  technique,  and  regulatory 
fulfillment.  It  is  the  source  of  substantial 
delays  and  added  expenditures. 

Rework  time  is  another  resource¬ 
consuming  effort.  This  involves  corrections 
of  errors,  reformulation,  redesign,  etc.,  often 
following  customer  and  higher  level  review 
and  comment.  The  four  major  waste 
factors  detract  from  getting  things  done. 
They  cost  unnecessary  expenditures  and 
delay  product  and  service  delivery.  In  some 
cases  they  constitute  80  to  90  percent  of 
productive  time;  6  to  7  hours  out  of  a  typical 
work  day  to  produce  nothing.  This  topic  will 
be  covered  in  more  detail  later  in  the  text  but 
suffice  to  say  that  it  is  a  much  larger  issue 
than  most  management  is  aware  of.  The 


following  paragraphs  provide  a  hint  of  the 
importance  of  waste. 

Rows  5,  6,  and  7  are  connected.  The 
number  of  employees  is  multiplied  by  their 
average  compensation  to  arrive  at  the  cost  of 
labor  input.  This  is  total  compensation 
including  wages  and  salaries,  retirement 
contributions,  insurance,  et  al.  Note  also  the 
meaning  of  this  number.  In  the  framework 
of  what  was  discussed  earlier,  it  represents 
the  value  of  employee,  management,  and 
supervisory  input  providing  needed  public 
products  and  services— the  studies,  dredging 
management  and  other  services  provided  by 
the  COE.  Row  7  shows  a  $63,000,000 
adjusted  appropriation  for  the  year  1996  and 
a  $62,000,000  adjusted  figure  for  1997. 
Row  7a  is  payroll  adjusted  for  waste.  The 
idea  here  is  that  due  to  waste  100  percent  of 
work  time  is  not  actually  provided  and 
detracts  from  real  payroll  productivity.  Time 
spent  on  transit  activities,  review,  rework, 
and  waiting  are  unproductive  dollars.  In  1997 
this  figure  is  shown  to  be  $7,000,000.  The 
dollar  figures  used  are  hypothetical  and  used 
for  illustration  only. 

Rows  9  through  14  represent  an  analysis 
of  capital  input  to  the  development  of  the 
multifactor  index.  Row  9  represents  the 
capital  stock  value  of  a  district— the  sum  total 
of  all  that  it  is  worth  in  terms  of  building, 
land,  equipment,  et  al.  This  was  mentioned 
earlier  as  the  stock  of  material  that  must  be 
converted  to  an  annual  flow  value.  To  make 
this  conversion,  the  value  of  the  stock  must 
be  depreciated  over  time  to  show  how  much 
of  each  type  of  asset  is  being  "used  up"  over 
time.  Technically,  this  depreciation  should  be 
done  by  asset  type  such  as  land,  buildings, 
equipment  and  other  fixed  and  movable 
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break-outs.  The  reason  for  this  is  that 
different  assets  depreciate  at  different  rates  (a 
building  may  depreciate  at  2  percent  a  year 
over  50  years  whereas  a  computer  may 
depreciate  at  20  percent  over  five  years).  An 
added  complication  is  that  new  equipment  is 
constantly  being  bought  and  mixed  with  older 
equipment  making  the  deployment  of 
depreciation  rates  more  difficult.  The 
example  here  is  simplified  so  as  not  to  get 
lost  in  the  forest  for  the  trees.  Depreciation 
represents  a  cost  of  doing  business;  assets  are 
used  up  and  their  worth  diminishes 
representing  a  cost.  They  must  be  replaced 
over  time  with  new  assets.  The  overall 
depreciation  rate  used  here  is  estimated  at  6 
percent  per  annum.  There  are  methods  and 
sources  to  determine  a  more  accurate  figure. 
Row  14a  is  a  reflection  of  waste  expended  in 
unproductive  use  of  capital.  It  is  quite  similar 
to  waste  on  the  labor  input  side.  When  labor 
is  wasted  so  is  the  capital  used  to  support  it. 
Thus,  it  is  appropriate  to  deduct  capital  waste 
from  capital  productivity  to  reflect  its  true 
effect.  In  1997  waste  amounted  to  $500,000 
in  capital  used  in  unproductive  endeavors. 

Rows  12  and  13  are  a  little  more 
complex.  In  this  case  the  purchase  of  an 
asset  or  capital  good  is  an  investment.  The 
cost  of  that  investment  is  what  the  money 
expended  to  buy  it  could  earn  if  invested 
elsewhere.  In  this  case  if  the  $18,418,000 
were  invested  in  long  term  securities  it  would 
yield  a  10  (example  only)  percent  rate  of 
return  or  $1,820,000  in  annual  income -an 
imputed  return  of  money.  In  other  words,  if 
the  capital  item  purchased  cannot  earn  at 
least  10  percent,  invest  the  money  in  long 
term  market  securities.  Since  it  is  anticipated 
that  your  investment  can  exceed  this  rate  of 


return,  the  decision  is  to  proceed  with  the 
investment.  The  land,  equipment,  building 
are  then  expected  to  yield  a  return  greater 
than  10  percent  otherwise  it  would  not  be 
rational  to  make  an  investment  earning  less. 
If  the  building  is  being  leased,  its  rental  value 
can  be  used  as  a  substitute  for  annual 
mortgage  expenses.  If  it  is  owned,  then  an 
equivalent  building's  rental  value  can  be  used 
as  a  proxy  measurement  for  building  costs. 
This  is  what  economists  call  the  opportunity 
cost  of  capital.  The  concept  helps  assure  that 
the  most  efficient  expenditures  of  funds  are 
made.  This  too,  is  an  annual  figure.  What  is 
implicit  in  this  is  that  the  purchase  or 
investment  by  the  government  in  equipment, 
buildings,  and  land  (i.e.  a  district)  will  yield 
at  least  10  percent  to  the  general  public  in 
terms  of  social  good  of  water  resources 
projects. 

Row  14  provides  the  final  value  for 
capital  cost.  It  is  the  sum  of  annual 
depreciation  costs  ($1,105,000)  and  annual 
opportunity  costs  ($1,842,000)  or 
$2,947,000.  This,  like  the  labor  cost  cited 
above,  is  a  cost  of  doing  business.  It  is 
capital  stock  cost  converted  from  a  "lump" 
value  $18,418,000  into  an  annual  flow  value 
so  it  can  be  added  to  labor  cost.  The  concept 
is  no  different  than  converting  the  purchasing 
price  of  a  home-say  $100,000  (capital  cost) 
into  annual  mortgage  payments  (annual  cost 
flow)  of  say  $400  per  month  times  12  months 
or  $4,800  per  year.  The  same  analysis  holds 
true  for  the  1997  figures. 

Rows  15  through  19  will  be  skipped  for 
the  moment  so  that  the  final  multifactor  index 
can  be  finalized.  Recall  that  a  multifactor 
index  is  the  ratio  of  the  value  of  output  to  the 
value  of  all  inputs— the  definition  of 
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productivity  described  earlier.  We  developed 
only  two  inputs  in  this  case,  labor  and 
capital,  largely  to  simplify  the  analysis  but 
also  because  labor  and  capital  represent  the 
vast  majority  of  production  costs— about  70 
percent  at  the  district  level.  We  do  not  need 
to  account  for  all  factors  of  production  to 
arrive  at  a  performance  indicator.  As  long  as 
there  is  consistency  in  enumeration  and  the 
other  factors  of  production  (supplies,  energy, 
etc)  are  not  expected  to  dramatically  change, 
we  can  utilize  the  main  controllable  factors  in 
generating  a  performance  measure.  Our  final 
multifactor  index  for  1997  is  found  by 
dividing  the  sum  value  of  waste  adjusted 
payroll  and  waste  adjusted  capital  by  adjusted 
output..  That  is:  ($35,000,000  + 

$2,247, 000)/$63.000, 000  =  0.59.  Notice 
that  as  waste  decreases  the  numerator  of  the 
expression  increases  which  is  reflected  in  the 
index  increasing.  It  should  be  recognized 
that  the  maximum  value  of  this  index  in  1997 
in  this  case  would  be:  ($42,000,000  + 
$2,947, 000)/$63, 000, 000  =0.71.  Insight 
can  be  gained  by  examining  the  factors  that 
result  in  an  index  of  1.00.  The  question  is, 
why  is  there  a  difference  between 
$63,000,000  and  the  sum  of  labor  and  capital 
($44,947,000)  or  $18,053,000.  These  are 
both  valuable  things  to  know.  It  is  not  the 
level  of  the  index  value  that  is  of  particular 
importance.  It  is  the  change  from  period  to 
period  that  is  important.  The  multifactor 
value  for  1997  changed  to  0.63  from  0.59. 
It  rose  because  the  number  of  employees 
declined  and  waste  decreased  substantially, 
driving  down  the  cost  of  labor  to  a  greater 
extent  than  output  declined.  The  reader  can 
now  explore  what  drives  productivity  up  and 
what  drives  it  down. 


Rows  15  through  19  contain  valuable 
performance  information  most  of  which  are 
spin-off  values  from  calculating  the 
multifactor  index.  Row  15  shows  output  per 
employee  which  is  a  partial  productivity 
index.  Notice  that  the  numerator  is  waste 
adjusted.  This  index  would  also  increase  as 
waste  decreases.  It  can  be  tracked  over  time 
and  benchmarked  with  others  for 
comparison. 

Row  16  shows  the  amount  of  capital 
used  by  each  employee.  The  adjusted  or 
unadjusted  ratio  can  be  used  depending  upon 
what  the  user  wants  to  use  the  information 
for.  The  question  here  is  what  capital 
improvements  can  be  made  to  improve 
productivity?  Clearly,  we  know  that 
computers  have  done  a  great  deal  to  improve 
productivity  during  the  last  twenty  years. 
They  can  improve  multifactor  performance 
by  driving  down  the  cost  of  doing  business— 
allow  us  to  do  more  with  less.  Total  cost  may 
increase  by  purchasing  more  productive 
capital  but  cost  per  employee  should  go 
down.  The  relationship  of  capital  to  labor 
can  be  explored  with  the  use  of  this  ratio. 

Output  per  hour  is  also  a  very  useful 
analytical  tool.  It  addresses  the  amount  of 
output  generated  by  each  hour  of  process 
time.  It  can  also  be  used  to  determine 
performance  improvement  over  time  and  for 
benchmark  comparisons. 

Output  per  square  foot  of  building  space 
(row  18)  is  useful  to  determine  the  efficiency 
of  space  utilization.  A  good  deal  of 
comparative  data  is  available  on  this  factor. 
It  is  a  frequently  used  index  and  a  measure  of 
efficiency  and  productivity. 

Row  19  is  the  ratio  of  output  to  capital 
discussed  earlier.  Again  the  adjusted  or 
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unadjusted  versions  can  be  used.  It  shows 
the  partial  productivity  of  output  relative  to 
capital  input  only.  Since  the  capital  cost 
involved  in  this  example  is  small  compared  to 
labor  input  costs  it  has  a  small  impact  on 
productivity. 

Column  4  has  been  ignored  until  now. 
Notice  that  the  values  in  this  column  are 
generated  by  dividing  the  measures  in  the  out 
year  by  those  in  the  base  year.  This  forms 
many  of  the  ratio  indexes  discussed  earlier 
and  some  of  the  partial  productivity  measures 
discussed  as  well. 

The  table  is  a  very  comprehensive  look 
at  an  organization's  over-all  performance 
with  important  detailed  information.  The 
data  can  be  used  for  year  to  year  comparisons 
or  can  be  benchmarked  against  other  districts 
(division,  offices,  branches,  et  al).  It  shows 
the  formation  of  many  of  the  performance 
measures  used  by  the  private  sector.  It  is 
designed  to  show  the  contribution  of  a 
number  of  resources  to  the  expenditures  of 
appropriations.  In  a  public  sector  context,  it 
is  very  useful  in  analyzing  cost  and  as  a  cost 
control  monitor.  Variations  of  the  tool  are 
possible  such  as  in  monitoring  overhead  in 
relation  to  output.  Output  need  not  always  be 
expressed  in  financial  terms.  It  could  just  as 
well  have  been  expressed  in  physical  terms 
such  as  the  number  of  reports,  cubic  yards  of 
dredging,  recreation  visitors,  etc.  Since  it  is 
difficult  to  express  white-collar  output 
(services)  in  tangible  terms  it  was  left 
portrayed  as  a  value-added  service. 

Factors  that  affect  productivity  usually 
include  training,  employee  experience 
(knowledge),  capital,  technology,  innovation, 
and  management-especially  management  of 
processes.  Knowledge  is  now  believed  to  be 


one  of  the  most  important  tools  for  increasing 
productivity  of  the  service  worker  [Drucker, 
1991],  The  impact  of  each  of  these  will 
depend  upon  the  organization  and  the 
customers  served.  "Productivity  is  not 
working  harder  or  longer,  but  working 
smarter"  [Drucker,  1991]. 

Simple  Aggregate  Indexes 

The  simple  aggregate  index  takes  two 
forms:  the  cost  index  and  the  quantity  index. 
The  simple  aggregate  cost  index  is  one  way 
of  dealing  with  the  problem  of  inflation 
mentioned  earlier.  It  involves  adding  up  all 
costs  for  each  item  included  in  the  cost  index 
during  each  succeeding  period  and  dividing 
each  of  the  sums  by  the  sum  of  the  process 
in  the  base  year.  For  example,  Table  13a 
shows  three  costs  items  (land  purchases, 
concrete,  and  labor)  used  in  the  construction 
of  a  flood  control  project  valued  in  the  base 
year  1994. 

If  C0  represents  the  cost  of  an  item  in 
the  base  year  and  Cn  the  cost  in  a  given  year 
for  which  we  are  evaluating  an  index 
number,  the  simple  aggregate  index  number 
is  equal  to: 

I  =  ECn/£C0  f°r  cost  analysis 
I  =  EQn/EQo  for  quantity  analysis 
where: 

I  =  Index  value 
C0  =  Cost  in  base  period 
Cn  =  Cost  in  period  n 
Q0  =  Quantity  in  base  period 
Qn  =  Quantity  in  period  n 
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The  simple  quantity  index  needs  special 
attention.  The  relative  influence  of  the 
quantity  items  in  the  index  are  not  tied  to 
their  relative  importance  and  the  units  of 
measurement  are  intermixed-the  apples  and 
oranges  dilemma.  Note  that  cubic  yards  of 
concrete  cannot  be  added  to  per  hour  labor 
cost  or  land  cost  per  acre.  Each  item  is 
essentially  given  the  same  weight  but  clearly 
some  of  the  values  can  be  much  larger  than 
others.  The  influence  is  effectively  inherent 
in  the  units  in  which  items  are  quoted.  This 
approach  clearly  has  limited  use.  If  the  items 
being  indexed  are  of  the  same  units  and  order 
of  magnitude  is  not  important,  this  is  an 
excellent  tool.  Units  are  being  mixed  and 
orders  of  magnitude  are  not  being  reflective 
of  relative  item  importance  to  the  total  cost. 

Simple  of  Average-of-Relative  Index 

It  is  possible  to  eliminate  the  influence 
of  units  in  an  index  by  first  calculating  the 
relative  cost  for  each  item  and  then  finding 
the  average  of  relatives.  This  is  illustrated  in 
Table  14a.  The  equation  for  this  is  expressed 
as: 

I  =  £(C0/Cn)/N  for  cost  analysis 
I  =  I(Q0/Qn)/N  for  quantity  analysis 
where: 

I  =  Index  value 
C0  =  Cost  in  base  year 
Cn  =  Cost  in  year  n 
Q0  =  Quantity  in  base  year 
Qn  =  Quantity  in  year  n 
N  =  Number  of  observations 


While  this  approach  overcomes  some  of 
the  unit  cost  problems  experienced  earlier,  it 
is  encumbered  by  the  problem  of  equal 
weights  for  each  of  the  cost  items.  Both  the 
simple  aggregate  and  simple  average-to- 
relative  methods  cannot  contend  with  the 
problems  of  weights.  The  weights  do  not 
reflect  the  importance  of  the  cost  variables 
involved.  Alternatively,  each  cost  element  is 
being  weighed  equally  in  the  analysis.  These 
problems  can  be  overcome  by  use  of  a 
weighting  process. 

Weighted  Aggregate  Indexes 
(Laspeyres1  Indexes) 

The  weighted  aggregate  index  is 
designed  to  handle  problems  of  item 
importance  in  the  construction  of  index 
numbers.  The  most  popular  method  is  to  use 
the  cost  or  quantities  produced  during  the 
base  year.  Note  that  this  index  can  be  used 
to  evaluate  both  cost  factors  and  quantity 
factors.  In  this  manner  the  units  in  which  the 
costs  or  quantity  variables  are  being 
measured  have  no  impact  on  the  index.  This 
technique  is  known  as  the  Laspeyres '  index 
and  the  basic  expression  for  this  index  is: 

L=  B(CnQ0/C0Q0)]  x  100 
the  cost  index  (base  year  quantity 

wts) 


L=  EKQ„C0/QoC0)]  x  100 
the  quantity  index  (base  year  cost 

wts) 
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TABLE  14a 

SIMPLE  AVERAGE-OF-RELATIVES  INDEX 


1  1 

Year 

Land  Costs 

Concrete 

Costs 

Labor  Costs 

Total  Costs 

Index 

Number  || 

|  1994 

100.0 

100.0 

100.0 

300.0 

100.0  | 

1  1995 

108.0 

102.9 

102.2 

313.1 

104.4  ; j 

|  1996 

128.0 

106.4 

97.8 

333.2 

111.1 

where: 

L  =  Index  value 
Cn  =  Cost  in  out  year  n 
C0  =  Cost  in  base  year 
Qn  =  Quantity  in  out  year  n 
Q0  =  Quantity  in  base  year 

Table  15a  demonstrates  how  to  apply 
this  index.  In  this  example,  project 
productivity  will  be  explored  using  inputs 
and  the  fixed  quantity  version  of  the  index. 
The  example  presents  energy,  capacity,  and 
labor  activity  measured  as  inputs  in  kilowatt- 
hours  (kWhrs),  kilowatts  (kW),  and  work 
hours,  respectively.  Notice  that  this  index  is 
a  composite  ratio  in  that  it  can  provide  a 
more  complete  picture  of  what  is  going  on 
because  it  takes  into  account  the  diverse 
characteristics  involved  in  measuring 
complex  interactions.  Simple  ratios  cannot 
simultaneously  consider  any  more  than  one 
characteristic  at  a  time.  The  KRA's  examined 


in  this  example  are  customer  care  and  cost 
per  unit;  two  entirely  different  items.  The 
background  issue  and  question  lies  in 
addressing  the  concern,  is  the  O&M 
provision  of  hydropower  to  the  community  a 
productive  endeavor?  The  question  must  be 
addressed  to  determine  whether  a  needed 
expansion  of  energy  for  future  use  should 
come  from  hydropower  or  from  coal  fired 
plants.  The  required  energy  facilities  (plant 
and  equipment)  can  be  constructed  for  about 
the  same  costs  (added  turbine  vs  a  coal  plant 
expansion).  The  operation  and  maintenance 
cost  is  the  deciding  variable.  The 
hydropower  operation  is  divided  into  three 
operating  components:  for  energy  generation 
it  is  the  O&M  cost  per  kW/hr,  for  capacity 
maintenance  it  is  total  cost  per  kW  of 
capacity,  and  for  labor  cost  it  is  the  cost  of 
labor. 

Column  (1)  of  Table  15a  shows  that  we 
are  examining  energy  costs  over  time  for  a 
fixed  amount  of  energy- 100, 000, 000  kWhrs. 
This  is  done  so  that  we  can  check  the  cost  of 
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energy  generation  over  time  for  a  base  year 
fixed  quantity.  This  is  no  different  than 
comparing  what  a  house,  automobile,  or 
groceries  of  the  same  quantity  used  to  cost  in 
the  "good  old  days".  Note  that  the  O&M 
cost  per  kWhr  is  increasing.  If  we  multiply 
this  constant  amount  of  energy  by  increasing 
cost  per  unit,  then  we  arrive  at  a  total  cost 
for  each  year  for  maintaining  and  operating 
energy  output  (excluding  labor). 
Immediately  the  ratio  of  out  year  to  base  year 
can  be  determined  as  shown  by  the  increasing 
values  from  100.0  in  1994,  to  110.0  in  1995, 
and  to  130.0  by  1996.  This  is  the  ratio  (over 
time)  index  discussed  earlier.  It  shows  that 
the  cost  of  providing  energy  has  increased  by 
10%  in  1995  and  by  30%  by  1996  compared 
to  the  base  period  of  1994. 

Column  (2)  shows  essentially  the  same 
process  but  is  applied  to  capacity  cost  over 
time.  Capacity  is  fixed  at  10,000,000  kW 
and  the  cost  to  maintain  this  capacity  also 
increases.  If  we  multiply  this  constant 
bundle  of  power  by  the  increasing  costs  per 
unit,  then  costs  are  observed  increasing  from 
year  to  year  for  capacity  as  they  were  for 
energy.  The  time  driven  ratio  shows  the  rate 
of  cost  increase. 

Column  (3)  adds  the  labor  component  to 
hydropower  costs.  It  is  evaluated  in  the  same 
manner  as  were  energy  and  power  with  the 
ratio  indicating  an  increasing  rate. 

To  arrive  at  an  aggregate  productivity 
index  for  all  three  factors  simultaneously,  we 
add  all  the  products  as  shown  in  Column  (4). 
Note  that  the  size  of  the  numbers  are  weighed 
by  their  relative  importance.  Energy  and 
labor  dominate  the  figures,  and  this  is 
reflected  in  the  totals.  The  more  significant 
numbers  have  the  largest  impact  on  the 


e> 


totals.  If  the  ratio  of  the  base  period  (1994) 
sum  in  Column  (4)  is  divided  into  the  out 
year  values,  then  an  aggregate  index  for  all 
three  factors  is  created.  The  index  values  in 
Column  (5)  shows  that  prices  are  increasing 
in  total.  An  examination  of  the  values  in 
Column  (1)  reveals  that  energy  is  the 
dominate  factor  because  it  is  made  up  of 
large  values  and  because  its  rate  of  growth  is 
also  the  most  significant. 

Five  important  ideas  are  represented  in 
Table  14a.  First,  the  units  problems  is 
avoided.  The  factors  in  the  weighing  process 
turn  all  the  units  into  total  costs.  Second,  the 
size  problem  is  avoided  because  each  factor 
is  weighted  according  to  its  relative  size. 
Third,  all  original  data  are  retained  so  that 
each  factor  can  be  examined  individually 
for  its  effect  on  the  total  index;  nothing  is 
lost  in  computing  the  aggregate  index. 
Fourth,  the  tables  illustrated  the  use  of  three 
performance  measures  at  one  time.  There  is 
nothing  to  prevent  the  user  from  employing  5, 
6,  or  10  aggregate  measures  in  one  table. 
One  last  matter  of  utility  is  that  this  type  of 
tabulation  can  be  provided  by  each  district  to 
its  division  office  where  they  can  be 
combined  to  represent  a  division  total.  This 
is  an  excellent  way  to  compare  performance 
between  COE  divisions  and  a  potential 
benchmark  setter. 

Table  16a  represents  another  aggregate 
index  evaluated  in  the  same  manner  as  in  the 
previous  example.  It  is  a  weighted  output 
index  using  fixed  output  quantities. 

Table  17a  shows  this  process.  If  we 
divide  the  output  and  input  aggregate  indexes 
from  Tables  15a  and  16a  respectively,  we 
arrive  at  an  index  reflecting  the  definition  of 
productivity:  output/input.  Used  in  this 
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TABLE  17a 

FINAL  OUTPUT/INPUT  RATIO  INDEX 

j  ( YEAR 

INPUT 

INDEX 

OUTPUT 

INDEX 

FINAL 

PRODUCTIVITY  || 
INDEX 

||  1994 

100.0 

100.0 

100.0  II 

||  1995 

106.4 

136.4 

128.  1  || 

||  1996 

117.2 

164.2 

140.1  || 

manner,  we  observe  that  the  new  ratio  goes 
from  100.0  in  1994,  to  128.3  in  1995,  and 
finally  140.1  by  1996.  This  shows  that 
hydropower  output  is  growing  faster  relative 
to  O&M  costs  over  time.  Hydropower 
productivity  is  thus  increasing  over  the  three 
year  period.  To  get  the  most  from  this 
analysis  we  would  have  to  conduct  the  same 
calculations  for  the  coal  fired  alternative  as  a 
reference  or  benchmark-the  original 
comparison  question. 

The  example  provided  in  Tables  15a, 
16a,  and  17a  employed  weighted  aggregate 
index  methodology  to  evaluating  the 
performance  of  a  project.  This  project 
consisted  of  an  O&M  hydropower  operation. 
It  could  have  just  as  well  been  a  flood 
control,  navigation  or  other  mission  activity. 
Table  18a  uses  the  same  techniques  for 
evaluating  program  performance.  In  this 
example  only  one  table  is  used  showing  the 


weighted  aggregate  program  productivity 
index  ( using  fixed  quantities )  without  the 
input/output  aspects  as  with  previous 
examples.  The  three  attributes  being 
evaluated  in  this  case  are  schedule  slippages, 
unspent  obligations,  and  late  contractor 
payment  penalties.  The  KRA  for  these  three 
attributes  are  tied  to  program  performance. 

The  "SCHEDULE  SLIPPAGES" 
performance  attribute  shows  that  12  months 
worth  of  slippages  costs  at  least  $100,000  per 
slippage  (average  start  up  expenses,  carry¬ 
over  costs,  etc.)  and  will  cost  the  taxpayers 
$1,200,000  in  1995.  In  1996,  this  same  12 
month  slippage  increases  in  value  due  to 
salary  increases  of  $110,000  yielding  added 
costs  of  $1,320,000  in  tax  burden.  By  1998 
and  1999,  added  costs  climb  to  $1,380,000 
and  $1,440,000,  respectively.  The  simple 
time  ratio  index  alone  shows  what  is 
happening.  Costs  relative  to  the  base  year 
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have  increased  by  10%  in  1996,  15%  in 
1997,  and  20%  by  the  year  1998.  A  similar 
analysis  is  done  for  "UNSPENT 
OBLIGATIONS"  and  "LATE 

CONTRACTOR  PAYMENT  PENALTIES" . 
The  dollar  cost  totals  are  added  horizontally 
and  divided  by  the  base  year  sum  to  provide 
the  aggregate  index  value  in  column  (5).  The 
"SCHEDULE  SLIPPAGES"  problem 
overrides  the  other  attribute  resulting  in  an 
index  value  that  is  gradually  rising  from 
100.0  in  1995  to  103.0  in  1998  despite 
decreasing  "LATE  CONTRACTOR 
PAYMENT  PENALTIES"  The  previous 
example  was  designed  to  demonstrate  three 
important  factors.  One,  the  weighted 
aggregate  productivity  index  can  be  applied 
to  program  analysis.  Secondly,  a  dominating 
attribute  such  as  "SCHEDULE  SLIPPAGES" 
will  out-weigh  a  rapidly  declining  cost  such 
as  "LATE  CONTRACTOR  PAYMENT 
PENALTIES" .  Last,  an  input  and  an  output 
table  are  not  both  needed  to  form  a  useful 
performance  measure.  Tables  19a 

and20a  show  yet  another  application  of  the 
weighted  aggregate  index  approach  to 
performance  measurement.  The  goal  in  this 
case  is  organizational  performance  (as 
opposed  to  project  or  program  performance). 
The  details  are  left  to  the  reader,  but  two 
things  are  important  to  note  in  Tables  19a 
and  20a.  One  is  that  the  index  applies 
equally  well  to  evaluating  organizational 
performance.  Two,  notice  that  waste, 
rework  and  error  in  Table  19a  are  increasing 
from  1995  to  1996,  then  begin  to  decline  in 
1997.  Table  20a  shows  the  reason. 
Increased  management  effort  in  controlling 
these  problems  through  the  use  of  quality 


control  charts,  personnel  training,  and 
review  time  have  impacted  the  downward 
direction.  Increased  proactive  management 
efforts  have  caused  an  impact;  and  it  is 
measurable  with  the  use  of  this  index.  The 
adverse  output  has  been  reversed  by 
increased  corrective  inputs  revealing  the 
efficacy  of  this  tool  to  show  a  cause  and 
effect  link. 

Table  21a  shows  the  relative  impact  of 
dividing  the  above  calculated  output  and 
inputs  (Tables  19a  and  20a)  to  arrive  at  a 
final  productivity  index.  It  is  taking 
relatively  more  corrective  dollar  input  effort 
to  get  a  dollars  worth  of  process 
improvement.  The  effort  is  paying  off 
because  the  magnitude  of  the  decreases  in 
waste,  rework,  and  error  outweighs  costs. 

Tables  22a  and  23a  are  similar  to  other 
weighted  aggregate  productivity  tables  with 
the  exception  that  costs  are  fixed  rather  than 
quantity.  In  this  situation  the  analyst  can 
observe  the  effects  of  varying  quantities  over 
time.  In  some  circumstances  costs  are 
relatively  stable  and  insensitive  relative  to 
quantity  variances.  That  is  a  5%  change  in 
quantity  has  a  greater  impact  on  total 
expenditures  than  a  5%  change  in  costs. 
This  is  a  phenomenon  economists  call 
elasticity.  It  occurs  when  costs  are  not 
directly  proportional  to  quantities  purchased 
or  used.  It  is  a  situation  frequently 
experienced  in  cost  estimating  although  most 
cost  estimators  are  not  always  aware  of  the 
concept.  This  form  of  productivity 
measurement  is  broadly  used  by  many 
government  agencies  and  the  private  sector  of 
the  U.S.  economy. 
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TABLE  21a 

FINAL  PRODUCTIVITY  INDEX 


. 

|  YEAR 

OUTPUT 

INDEX 

INPUT 

INDEX 

FINAL 

productivity! 

INDEX 

1995 

100.0 

100.0 

100.0 

j  1996 

110.2 

121.2 

90.9 

j  1997 

109.3 

140.5 

77.8  | 

The  Consequences  of  Declining 
Productivity 


Increasing  productivity  has  the  opposite 
impact: 


Many  public  institutions  believe  they  are 
immune  or  cushioned  from  the  consequences 
of  declining  productivity;  to  the  point  of  not 
even  measuring  them.  This  leads  to 
catastrophic  results  in  the  long  run.  Today, 
with  the  enormous  interest  in  productivity 
and  efficiency,  this  philosophy  is  beginning 
to  take  its  toll.  Declining  productivity  is  a 
deadly  trap!  The  cause  and  effect 
consequences  are  strong  and  clear.  The 
spiral  goes  like  this: 


Productivity  decreases(-)  =  Cost  per  unit 
output  increases(-)  =  Less  competitive(-)  = 
Customers  fade  away  (-)  =  Need  declines(-) 
=  Adverse  impact  on  capital,  materials, 
employees(-)  =  Further  declines  in 
productivity . an  so-on. 


Productivity  increases! +)  —  Cost  per  unit 
output  decrease! +)  =  More  competitive! +) 
=  Customer  demand  grows! +)  =  Need 
increases! +)  =  Positive  impact  on  capital, 
materials,  employees! +)  =  Further 

improvements  in  productivity. 


The  effects  of  changes  in  productivity  do 
not  stop  with  productivity  itself  but  spill  over 
onto  efficiency  and  effectiveness;  our 
performance  measures  and  the  target  of 
GPRA.  Government  institutions  do  not 
realize  that  the  world  has  changed  over  the 
last  two  decades.  The  world  is  finding  more 
and  more  substitutes  for  public  works: 
contracting  out,  customers  doing  it 
themselves,  more  subsidies  of  capital  for 
labor,  foreign  competition,  etc. 
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A.2.  KNOWLEDGE 
PRODUCTIVITY 


Over  25%  of  the  growth  in 
productivity  in  the  U.S.  from 
1948  to  1981  came  from 
increases  in  education  per  worker  [BLS, 
1983].  This  is  a  significant  factor  and  should 
not  be  down-played.  The  usual  response  of 
management  to  low  productivity  growth  is  to 
employ  more  labor  saving  capital  or  work 
employees  longer  and  harder.  This  is  no 
longer  the  best  response  to  the  problem. 

"In  fact,  knowledge  is  the  only 
meaningful  resource  today.  The  traditional 
"factors  of  production"— land  (i.e.  natural 
resources),  labor,  and  capital— have  not 
disappeared,  but  they  have  become 
secondary.  They  can  be  obtained  and 
obtained  easily,  provided  there  is  knowledge. 
And  knowledge  in  this  new  sense  means 
knowledge  as  utility,  knowledge  as  the  means 
to  obtain  social  and  economic  development" 
[Drucker,  1990]. 

Given  the  importance  of  this 
characteristic  of  productivity,  it  may  be  wise 
to  develop  and  utilize  a  knowledge 
productivity  index.  Knowledge  in  the 
context  of  how  it  is  being  used  today  is 
defined  to  include:  education,  training,  skill 
development,  experience,  information 
gathering,  and  information  analysis  and 
distribution.  To  be  useful,  knowledge  must 
not  only  be  "mechanical"  in  nature  but 
"tacit"  as  well.  Tacit  knowledge  is  the  ability 
to  merge  academic  training  and  experience  to 
synthesize  new  knowledge.  It  is  part  of  the 
"information  highway"  and  "cyberspace". 
Table  24a  illustrates  some  of  the  more  useful 
measures  of  knowledge.  Notice  that  the 


indicators  can  be  group  averaged  or 
presented  as  an  over-all  average. 

A  knowledge  "quick  check"  is 
demonstrated  as  follows:  knowledge 
capability  =  average  years  formal  schooling 
x  average  years  of  professional  experience  x 
no.  of  FTE’s  or  (example:  15.6  average 
years  formal  schooling  x  15.6  average  years 
professional  experience  x  400  FTE’s  = 
102,960  knowledge  years).  It  contains  less 
detail  than  the  tabulated  data  but  is  none-the- 
less  a  fairly  good  indicator.  The  indicator 
blends  the  effect  of  academic  training,  career 
experience,  and  the  number  of  personnel  into 
one  measure. 

These  can  be  used  to  "benchmark" 
against  those  organizations  which  are  noted 
to  be  the  best  in  a  particular  product  or 
service  as  a  guide-post.  How  are  the  leaders 
in  business  using  knowledge  to  increase 
productivity,  efficiency,  and  effectiveness? 
Comparisons  of  knowledge  indicators  against 
others  and  over  time  may  yield  some 
valuable  information. 

Efficiency 

Efficiency  is  another  simple  but  useful 
measure  of  performance.  [Stevenson,  1993]: 

Efficiency  =  Actual  Output/Effective 
Capacity 

Effective  capacity  is  the  maximum 
amount  of  output  given  an  activity  or  process 
mix,  scheduling  problems,  equipment 
maintenance,  quality  parameters,  etc. 
[Stevenson,  1993].  Productivity  and 
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efficiency  are  often  confused.  Efficiency 
pertains  to  getting  the  most  out  of  a  given  set 
of  resources  [Stevenson,  1993].  Productivity 
pertains  to  effective  use  of  all  resources  and 
is  a  narrower  concept  than  efficiency.  It  is  a 
subset  of  efficiency.  It  has  been  described  as 
the  ratio  of  resources  expected  to  be 
consumed  to  resources  actually  consumed 
[Sink,  1985],  For  example,  efficiency  would 
be  described  as  using  an  engineer  to  draft 
engineering  drawings  given  a  drafting  table, 
compasses,  triangles,  etc.  If  this  is  the  only 
resource  available  to  do  that  task,  then  it  is 
an  efficient  operation.  Productivity  would  be 
realized  by  using  a  technician  to  do  the  same 
task,  working  with  a  CAD  system.  An 
example  of  an  efficiency  calculation  would 
go  as  follows: 


The  effective  capacity  of  a  new  FAX 
machine  is  five  pages  per  minute.  The  same 
FAX  machine  in  your  office  is  three  years 
old  and  typically  operates  at  three  pages  per 
minute.  The  efficiency  of  your  machine  is 
down  to:  3  pages/5  pages  =  .60  or  60%. 

Table  25a  provides  additional  examples 
of  calculating  efficiency  for  different 
situations.  If  these  ratios  are  under  1.00  this 
is  an  indication  that  performance 
improvements  are  warranted.  They  serve  as 
indicators  and  diagnostic  tools  for  motivating 
change.  As  the  examples  indicate,  efficiency 
can  be  applied  to  project  performance  or 
organizational  business  practices.  This  is  the 
case  for  all  performance  indicators  presented 
in  this  report. 
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TABLE  24a  (Continued) 
KNOWLEDGE  PERFORMANCE  INDEXES 
(In-House  Attributes  Only)  Last  Years  Figures 


Knowledge  Type 


5.  Avg.  Printer  Age 


6.  Avg.  Software  Age 


Professional  Experience/Skill  Level 


1.  0-5.9  Years 


2.  6.0  -  10.9  Years 


3.  11.0- 15.9  Years 


4.  16.0  -  20.9  Years 


5.  21.0-24.9  Years 


Measure 


Months 


COE 

Office1 

(1) 


Comparison 

Benchmark 

(2) 


Ratio 

(l)/(2) 

(X100) 


1  Columns  (2)/(l)  for  items  5  and  6 
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TABLE  25a 

EXAMPLES  OF  EFFICIENCY  CALCULATIONS 


Actual  Output 

Calculated 

Effective  Capacity 

Efficiency  (X  100) 

Eight  reports  generated  by  a 
staff  of  30  in  a  particular  branch 
for  FY  97. 

12  Reports 

8/12  =  66.7 

800  man-hours  to  design  a 
spillway  for  a  dam. 

700  Man-Hours 

800/700  =  114.3 

1,000,000  annual  visitors  to  a 
dam  and  reservoir  in  1999. 

1,100,000  Visitors 

1,000,000/1,100,000  =  90.9 

1,000  Kwhrs  generated  by  a 
hydro-power  plant  in  January. 

1250  Kwhrs 

1000/1260  =  80.0 

4  wolves  per  10,000  acres  in  a 
region  of  woods. 

6  Wolves 

4/6  =  66.6 

Removal  of  100,000  cy  of 
material  from  a  shallow  draft 
channel  requires  200  man¬ 
hours;  500/cy/man-hr. 

550  Cy/man-hr 

500/550  =  90.9 

$500,000  in  overhead  expenses 
for  a  district  of  400  FTE’s  in 
the  month  of  March. 

$400,000 

500,000/600,000  =  83.3 

Mailroom  cost  $350,000  a  year 
to  operate. 

$275,000 

275,000/350,000  =  78.6 

A  district  inspected  30  dams  in 

FY  1998. 

25  Dams 

30/25  =  120.0 

It  cost  a  district  $0.05  per  page 
to  print  200  volumes  of  a  150 
page  report. 

$0,053 

0.053/0.05  =  166.0 
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A.3.  DEFLATING  DATA 


0  J  Q  J  J 


The  dollars  used  to  estimate  costs 
per  unit,  expenditures,  budgets, 
values,  and  revenues  for 
performance  indexes  do  not  retain  their  base 
year  purchasing  power  during  periods  of 
rising  costs  or  during  periods  of  deflation. 
In  order  for  an  analyst  to  determine  what  a 
given  amount  of  dollars  will  purchase 
relative  to  previous  years,  adjustments  are 
necessary  to  arrive  at  real  purchasing  power. 
There  are  a  number  of  prepared  indexes 
periodically  published  by  public  and  private 
institutions  which  can  be  used  to  adjust  for 
changes  in  the  value  of  the  dollar  over  time. 
The  Bureau  Of  Labor  Statistics  (BLS),  the 
Engineering  News  Record  (ENR),  and  the 
Federal  Reserve  Board  (FRB)  are  three 
frequently  used  sources.  There  are  also 
specific  indexes  designed  for  different 
purposes  such  as  those  used  to  adjust  for 
wages,  heavy  construction  costs,  light 
construction,  real  estate,  the  consumer  price 
index,  wholesale  price  index,  et  al.  The 
analyst  must  choose  the  one  best  suited  for 
his  or  her  specific  purpose.  You  would  not 
use  a  heavy  construction  index  to  adjust  real 
estate  prices,  for  example. 

During  inflationary  periods  prices  or 
costs  increase  and  erode  dollar  values. 
Alternatively,  during  periods  of  recession, 
the  purchasing  value  of  the  dollar  tends  to 
climb.  The  following  example  will  illustrate 
the  principles  and  methods  involved. 

Suppose  that  a  district's  annual  budget  is 
increased  from  $125,000,000  to 


$135,000,000  over  a  one-year  period  when 
the  ENR  cost  indexes  change  from  a  relative 
value  of  90  to  a  relative  value  of  125.  What 
can  be  said  about  the  $10,000,000  increase  in 
budget? 

The  ratio  of  the  new  budget  to  the  old 
budget  is:  $135,000,000/$125,000,000  = 
1.08  or  108.0%.  The  dollar  value  of  the 
budget  went  up  by  8%.  When  adjusted  for 
inflation  using  the  ENR  index  the  new  real 
budget  is:  $125,000,000/125  = 

$108,000,000.  The  old  budget  is  actually: 
$125,000,000/90  =  138,888,900.  The  ratio 
of  the  two  real  budget  changes  is: 
$108,888,900/138,888,900  =  .778  or  77.8% 
which  represents  a  real  decrease  in 
purchasing  power.  The  dollars  provided 
today  are  only  able  to  buy  77.8%  of  what 
they  bought  a  year  earlier.  Even  though  the 
money  value  of  the  budget  has  increased  by 
$10,000,000  over  the  period  the  real  or 
purchasing  power  has  actually  declined.  In 
other  words,  the  $135,000,000  now  buys 
77.8%  of  what  the  old  $125,000,000 
procured  in  a  previous  year.  Table  26a 
shows  another  example  of  how  to  deflate 
dollars  (costs,  budgets,  expenditures,  etc.). 
Notice  that  inflation  over  time  has 
deteriorated  the  purchasing  power  of 
increasing  dollar  amounts  (column  920  vs. 
column  (4)).  This  is  because  inflation  has 
risen  faster  than  budgets.  By  the  year  2001, 
the  budget  has  risen  faster  than  inflation 
returning  the  purchasing  power  of  the  budget 
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TABLE  26a 

THE  DWINDLING  PURCHASING  POWER  OF  THE  DOLLAR 

(1982  -  84  =  100) 


ii  1 

!  a) 

(2) 

(3) 

(4) 

YEAR 

BUDGET 

COST  INDEX 

REAL  BUDGET 

|  1996 

$125,000,000 

158.0 

$79,113,924  | 

II  1997 

$128,000,000 

163.0 

$78,527,607  || 

||  1998 

$130,000,000 

166.1 

$78,266,105  || 

1  1999 

$132,000,000 

175.4 

$75,256,556  |j 

v  >: 

II  2000 

$135,000,000 

187,8 

$71,884,984 

|1  2001 

£  £ 

$150,000,000 

189.6 

$79,113,924  || 

amounts  to  those  of  1996-the  comparative 
base  year. 

It  is  vital  to  adjust  the  dollar  values  of 
performance  indexes  that  use  price,  costs, 
dollar  values,  etc.  prior  to  using  them  to 
make  performance  indexes.  If  this  is  not 
done,  then  comparisons  of  performance  will 
be  distorted  and,  incorrect  decision  will 
likely  be  made.  The  unadjusted  index  will 
measure  relative  price  changes  and  not  actual 
productivity.  This  will  adversely  impact  the 
goals,  objectives,  strategic,  tactical,  and 
operational  characteristics  of  the  organization 


described  earlier.  It  will  likely  filter  through 
the  entire  system  undermining  the  fabric  and 
efficacy  of  decision  making.  Incorrect 
information  is  one  of  the  performance 
problems  GPRA  mentions  as  a  cause  of 
reduced  effectiveness  and  efficiency.  In  the 
face  of  performance  measurement,  this  is  not 
a  desirable  possibility.  To  make 
adjustments,  the  COE  will  also  need  to 
determine  a  base  year  as  an  anchor  for 
relative  comparisons.  A  study  of  the 
example  in  the  table  will  show  why  this  is  so. 
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A.4.  BASE  YEAR 
SELECTION  FOR 
PERFORMANCE  MEASUREMENT 


In  initiating  a  performance  measure¬ 
ment  program  care  must  be  taken  in 
selecting  the  base  year  to  which 
comparisons  of  future  performance  will  be 
calibrated.  The  base  year  for  indexes  or 
indicators  is  the  underlying  framework  for 
all  future  measurement.  If  an  inordinately 
active  year  is  selected  without  forethought, 
future  years  may  witness  a  decline  in 
numerical  value  without  or  despite  any  action 
taken  to  influence  or  to  control  the  outcome. 
The  consequences  are  also  true  if  an 
inordinately  inactive  year  is  chosen.  This  is 
best  illustrated  in  Figure  2a.  Suppose  that 
the  year  1994  is  selected  as  a  base  year  with 
activity  level  (1).  Note  that  this  year  is 
below  the  historic  performance  trend 
(represented  by  the  dotted  line)  for  a 
particular  index.  This  low  value  was  due  to 
some  anomaly  to  which  no  one  gave  a  second 
thought.  Management  is  now  prepared  to 
embark  on  improving  performance  with  the 
base  year  determined.  The  following  years 
will  witness  increases  in  performance  such  as 
the  year  1995.  Management  will  be  led  to 
believe  that  its  goals,  objectives  and  decision 
making  are  right  on  target-the  index  has 
increased.  However,  management  may  be  in 
for  a  surprise  because  the  selected  base  year 
was  well  below  par  to  begin  with  and  would 
have  likely  increased  in  any  event.  While  this 
performance  may  appear  to  be  cause  for 
celebration,  continuance  of  these  same  goals, 
objectives  and  decision  making  may  not  be 
able  to  stop  another  decline  in  future  years 


because  management  endeavors  were 
predicated  on  incorrect  notions  in  the  first 
place.  Such  a  situation  is  observed  in  the 
year  1996.  Now,  confusion  or  paralysis  sets 
in  and  a  fresh  approach  will  need  to  be 
developed.  This  all  could  have  been  avoided 
in  the  first  place  if  a  more  suitable  trend  line 
base  year  were  selected. 

The  year  1997  is  also  not  a  desirable 
base  year  selection  for  the  opposite  reasons. 
With  this  anchored  base  year  selected 
management  goes  on  to  develop  a  congruent 
set  of  goals,  objectives,  and  operating 
policies  to  embark  on  an  otherwise  sound 
performance  program.  Because  an 
inadvertently  active  year  was  chosen  as  the 
bases  to  measure  future  performance,  trouble 
arises  in  1998.  The  index  value  drops  from 
point  (3)  in  1997  to  point  (4)  in  1998.  This 
was  not  due  to  management  incompetence  but 
rather  the  selection  of  an  unusual  year. 
Confusion  sets  in  and  management  searches 
for  new  goals,  objectives  and  corrective 
actions  on  operating  procedures.  The 
antecedent  performance  plans  developed  in 
1997  were  perfectly  good  but  are  now 
incorrectly  discarded. 

Such  incidents  as  those  described  above 
are  possible  if  care  is  not  taken  to  seek  out  a 
viable  base  year.  The  costs  of  the  incorrect 
base  year  selection  are  high  and  can  be 
avoided.  Some  study  and  care  must  be 
undertaken  to  be  certain  that  the  base  year  is 
one  that  can  be  lived  with  in  the  future. 
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In  some  cases  the  values  of  previous 
year  numbers  are  subject  to  a  high  degree  of 
volatility  by  the  nature  of  the  variable  being 
measured  and  its  vulnerability  to  political 
decisions.  Deep  draft  navigation  tonnage  data 
is  a  case  in  point.  The  political  arena  in  the 
past  has  often  thrown  data  into  extreme 
directions  due  to  embargos,  blockades,  fear 
of  conflict,  etc.  There  are  tools  which  can  be 


employed  to  determine  or  detect  underlying 
trends  and  lead  to  wise  base  year  selections. 
These  tools  include:  moving  and  weighted 
averages,  regression  analysis,  exponential 
smoothing,  shift-share  analysis,  Box-Jenkins 
and  Auto  Regressive  Integrated  Moving 
Averages  (ARIMA)  techniques  and  a  host  of 
others. 
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A.5.  CYCLE  TIME  INDEX 
AND  VALUE-ADDED 
ANALYSIS 


u  As  a  strategic  weapon,  time 
/\  is  the  equivalent  of  money, 
x  M.  productivity,  quality,  and 
innovation "  [Stalk,  1988].  One  of  the  more 
important  performance  indicators  is  that  the 
cycle  time  index,  also  known  as  "Lead  Time" 
[Management  Concepts,  Inc.,  1995],  Cycle 
time  is  the  total  length  of  time  used  to 
complete  an  entire  project.  It  measures  how 
quickly  we  can  accomplish  our  goals  and 
objectives;  complete  our  projects,  functions, 
and  programs.  The  word  transformation  was 
used  earlier  in  this  report  and  described  as 
the  activities  we  do  from  the  resource  input 
stage  to  the  product  or  service  completion 
stage.  It  is  the  sequence  of  events  where 
people  and  equipment  "add  value"  to  raw 
materials  and  ideas  to  produce  an  output  and 
eventually  an  outcome.  This  is  called 
process  time.  This  is  how  iron  ore  becomes 
steel  which  ultimately  is  used  to  produce  a 
building  or  bridge.  Non-valued  activities 
such  as  rework  and  review  time  interfere 
with  the  transformation  process  and 
productive  activities  to  increase  cycle  time 
resulting  in  delays  in  customer  satisfaction 
and  increased  costs.  There  are  a  number  of 
non-value  added  elements  which  can  be 
measured  to  produce  a  cycle  time  index. 
These  include  transit  time  or  inter-activity 
time  (the  time  between  activities  or  the  time 
within  activities  when  nothing  gets  done- 
simple  waste),  wait  time  (waiting  for 
something  to  happen  before  proceeding),  and 
review  time  or  quality  control  time  (which 
encompasses  error  detection)  and  rework  time 


(the  time  used  to  correct  errors  or  make 
changes  required  by  review  authorities). 
More  will  be  said  about  these  issues  in  the 
paragraphs  that  follow. 

As  stated  earlier,  the  idea  behind 
continuous  process  improvement  is  to  "do  it 
right  the  first  time".  Build  quality  into  the 
systems,  not  inspect  it  out.  The  Cycle  Time 
Efficiency  index  is  written  as  follows: 

CTE  =  PROCESSING  TIME/ 
(PROCESSING  TIME  +  TRANSIT 
TIME  +  WAITING  TIME  + 
REVIEW  TIME  +  REWORK  TIME) 

For  example,  suppose  it  takes  12  months 
of  actual  processing  time  to  produce  a 
feasibility  report  in  a  planning  division.  The 
amount  of  time  that  report  components 
expend  undergoing  a  movement  from  one 
branch  to  another  and  waiting  for 
intermediate  inputs  such  as  H&H  analysis, 
cost  analysis,  etc,  is  determined  to  be  8 
months.  The  waiting  time  due  to  guidance 
clarification  and  local  sponsor  inputs 
including  meetings,  etc.  is  found  to  be  4 
months,  and  review  time  at  the  district  level 
is  another  4  months.  Post  report  re-work 
time  (report  returned  after  higher  level 
review  and  comment)  due  to  plan  formulation 
problems  and  the  addition  of  new  plan 
formulation  alternatives,  adds  another  12 
months.  The  cycle  time  efficiency  index  is: 
12/  (12  +  8  +  4  +  4  +  12)  =  0.30  or 
30.0% .  A  value  of  1 .00  for  the  index  is  the 
best  that  can  be  done.  Given  that  a  value  of 
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1.00  is  seldom  practical,  the  reader  can 
readily  see  the  value  of  the  index.  This 
analysis  can  be  done  for  nearly  every  activity 
in  an  organization;  including  planning, 
engineering,  program  performance, 
organizational  performance,  etc.  Keeping 
records  of  how  time  is  actually  used  is  a  vital 
part  of  process  time  improvement.  Recall 
also  that  the  term  “customer”  includes  both 
internal  and  external  clients. 

The  four  major  time  delay  types  listed 
above  can  be  broken  into  a  variety  of  smaller 
categories.  A  list  needs  to  be  developed  and 
standardized  in  some  detail  for  consistency 
between  organizational  elements  and 
consistency  over  time.  More  importantly, 
time  losses  can  be  converted  into  costs  as  a 
means  of  dramatizing  their  magnitude.  For 
example,  move  time  may  have  resulted  in 
unnecessary  expenditures  of  $50,000,  waiting 
time  $100,000,  inspection  time  another 
$75,000,  and  re-work  time  $200,000.  This 
yields  a  total  dollar  value  of  time  lost  of 
$425,000  in  addition  to  the  original  report 
processing  time( the  cost  of  doing  it  right  the 
first  time).  This  figure  represents  the  dollar 
value  of  time  lost.  It  is  often  simply  called 
direct  labor.  Waste,  expressed  in  terms  of 
both  time  lost  and  the  added  cost  of  doing 
business,  is  one  way  to  drive  home  the  need 
for  change.  The  performance  indicators 
developed  in  previous  paragraphs  can  also 
aid  in  detecting  and  locating  time  delay 
problems.  In  a  sense,  actual  time  lost  is  more 
damaging  because  it  cannot  be  recovered;  to 
an  extent  dollar  losses  can  be  replenished. 
Either  way  the  customer  pays  unnecessarily. 

Cycle  time  is  more  important  in  what  the 
COE  does  than  in  many  other  organizations. 
This  is  because  cycle  time  for  most  projects 


is  quite  lengthy  running  typically  17  years 
from  start  to  finish.  The  difficulty  with  long 
cycle  times  lies  in  the  capricious  events  that 
can,  and  do,  occur  during  the  transformation 
process.  For  one  thing,  long  cycle  times 
mean  a  good  deal  of  value-added  input  is 
accumulated  before  the  final  product  or 
output  (a  project)  is  available  to  the  public. 
This  opens  considerable  opportunity  for 
circumstances  to  change-sometimes 
radically.  Engineering  changes,  planning 
changes,  the  public's  perceptions  change, 
guidance  changes  et  al.  This  results  in 
many  restarts,  setbacks,  waiting  time, 
reanalysis,  chaos,  and  a  host  of  other  cost 
and  time  incurring  events.  It  can  lead  to  a 
loss  of  focus  as  employees  and  management 
search  for  a  new  sense  of  direction  for  the 
already  accrued  effort.  Work  in  progress  can 
simply  come  to  a  halt  requiring  a  new 
momentum  and  sense  of  direction.  Secondly, 
the  work  done  by  the  COE  is  technically 
complex  and  involved  compounding  the 
situation  especially  since  frequent  progress 
checks  are  necessary  to  keep  schedules  on 
track.  Third,  long  cycle  times  often  result  in 
unbalanced  workloads  and  production 
bottlenecks.  When  one  upstream  work  center 
completes  its  tasks  there  is  an  immediate 
influx  of  new  activities  for  those  next  in  line 
or  further  downstream  in  series  related 
activities.  This  leads  to  a  feast  or  famine 
work  environment.  Fourth  and  last,  is  that 
long  cycle  times  are  inherently  inflexible. 
Several  years  of  accumulated  work  is 
difficult  to  reconfigure  and  restart  without 
dramatically  impacting  time  schedules.  It  is 
a  costly  endeavor  as  well. 

Value-added  is  a  concept  very  closely 
linked  to  cycle  time.  Value-added  is  the 


Appendix  A  -  Performance  Measurement  -  84 


increase  in  the  worth  of  a  product  or  service 
as  it  progresses  from  an  idea  and  raw 
materials  into  a  completed  output  or  service 
with  attributes  customers  are  expecting.  The 
activities  that  take  place  between  these  two 
points  is  the  transformation  process  where 
value  is  added  by  employee  knowledge, 
work  efforts,  equipment,  and  materials  to 
complete  the  product  or  service.  As  each  step 
or  activity  progresses,  cost  and  value  accrue. 
At  any  point  within  the  transformation 
process  the  value  of  the  incomplete  product 
or  service  is  called  its  "book  value". 

Many  of  the  events  that  take  place 
during  the  transformation  process  more  often 
than  not  add  considerable  unnecessary  costs 
to  the  final  product  or  service  without 
providing  value  to  the  customer  or  require 
the  customer  to  wait  longer  for  the  delivered 
product  or  service..  These  events  are  not 
what  the  customer  wants  to  pay  for.  These 
unnecessary  tasks  and  activities  are  waste. 
Waste  can  also  be  viewed  as  "any  process 
that  increases  costs  but  does  not  add  value  to 
the  product"  [Masked,  1991].  It  is  one  of  the 
reasons  that  "95%  of  productive  activities  are 
non-value  adding"  [Masked,  1991].  This 
proportion  is  made  up  of  the  wealth  depleters 
mentioned  earlier;  transit  time,  waiting  time, 
review  time,  and  rework  time.  Instead  of 
shrinking  waste,  traditional  management's 
reaction  is  to  increase  productivity  by  forcing 
more  work  into  an  already  nominal 
processing  time  frame  (working  harder  and 
longer  rather  than  smarter).  This  usually 
results  in  adverse  conditions  and  a  stressed 
system. 

Table  27a  illustrates  the  concepts.  An 
employee  has  been  assigned  five  projects 
(studies,  assignments,  etc.)  to  work  on.  The 


table  shows  a  one  day  slice  of  time  out  of  the 
employee's  work  schedule.  During  that  day 
the  employee  is  able  to  spend  four  processing 
hours  (value  adding)  on  Project  A,  two 
processing  hours  on  Project  B,  and  two 
rework  hours  on  Project  E.  The  employee 
has  worked  a  full  eight  hour  day.  Notice, 
however,  that  the  remainder  of  the  projects 
have  spent  the  majority  of  time  in  non-value¬ 
adding  activities.  A  full  days  work  has  only 
contributed  to  six  hours  of  value-adding 
transformation.  In  total,  six  out  of  thirty-six 
hours  of  activities  were  productive;  83.3 
percent  of  potentially  value  adding  time  was 
wasted  on  tasks  which  did  not  contribute  to 
the  value  of  the  product  or  service  to  the 
customer. 

Table  28a  was  taken  from  [Harrington, 
1991]  and  provides  another  example  of  the 
way  waste  contributes  to  increased  cycle  time 
and  subsequently  increased  process  cost. 
These  are  not  atypical  situations  but  are 
none-the-less  a  profound  finding.  There  is 
plenty  of  ground  to  do  battle  with  cost 
reduction  and  cycle  time  savings.  The  work 
we  do  must  be  viewed  through  the  eyes  of 
the  intended  customer.  The  letter  processing 
example  may  show  that  productive  work  is 
being  done  by  the  secretary  while  she  waits 
for  review,  the  bosses  signature,  et  al,  but 
the  time  delay  is  experienced  by  the 
customer.  Both  the  index  and  the  supporting 
table  are  excellent  tools  for  diagnosis  and 
presentation.  They  are  the  targets  of  GPRA 
and  fulfill  APIC  mandates. 

It  is  recognized  that  non-value-added 
tasks  are  necessary  to  run  an  organization. 
The  necessary  ones  include  record  keeping, 
order  forms,  personnel  record  keeping, 
payroll  work,  budget  analysis,  etc.  These  are 
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TABLE  28  a 

PROCESSING  TIME  VERSUS  CYCLE  TIME 


Consider  the  cycle  time  of  a  letter-writing  process: 


Activity  v*  *:•  .f.-  -S  .  *:• 

Processing 

Time 

(Hours) 

Cycle 

(ittours) 

1. 

A  manager  estimates  that  it  takes  12  minutes  to  write  a  one-page  memo  and  place  it  in 
the  outgoing  basket. 

0.2 

0.2 

2. 

The  manager’s  secretary  picks  up  outgoing  mail  twice  a  day,  at  9  a.m.  and  1  p.m.  Average 
delay  time  is  12  hours. 

0.1 

12.0 

3. 

The  secretary  assists  three  managers,  answers  phones,  schedules  meetings,  processes 
incoming  mail,  retypes  letters,  and  performs  special  assignments.  All  these  activities  have 
priority  over  typing.  Average  time  before  starting  to  type  a  letter  is  26  hours. 

26.0 

4. 

The  secretary  types  the  memo  and  puts  it  into  the  manager’s  incoming  mail. 

0.3 

0.3 

5. 

Incoming  mail  and  signature  requests  are  delivered  twice  a  day;  at  9  a.m.  and  1  p.m. 

12.0 

6. 

The  manager  reads  incoming  mail  at  5  p.m. 

0.1 

17.0 

7. 

The  secretary  picks  up  mail  at  9  a.m. 

16.0 

8. 

Retyping  is  a  priority  activity  and  is  returned  in  the  1  p.m.  mail  delivery.  (Note:  60  percent 
of  all  letters  are  changed  by  managers.) 

0.2 

4.0 

9. 

The  manager  reads  and  signs  letters  at  5  p.m. 

0.1 

4.0 

10. 

The  memo  is  picked  up  by  the  secretary  at  9  a.m. 

0.1 

16.0 

11. 

The  memo  is  put  in  the  copy  file  and  held  for  the  next  trip  to  copy  center  at  2  p.m. 

0.1 

5.0 

12. 

The  secretary  walks  to  the  copy  center,  makes  copies,  and  addresses  envelopes. 

0.1 

.01 

13. 

The  secretary  takes  memo  to  mail  box  by  5  p.m. 

0.2 

2.5 

14. 

Mail  is  picked  up  at  8  a.m. 

15.0 

15. 

The  memo  is  held  in  the  mailroom  for  afternoon  mail  delivery  at  3  p.m. 

0.1 

17.0 

16. 

Secretary  2  picks  up  mail  and  sorts  at  4  p.m.  Secretary  puts  memo  in  manager’s  incoming 
mail. 

0.1 

1.0 

17. 

Secretary  delivers  incoming  mail  to  manager  2  at  9  a.m. 

0.1 

14.0 

18. 

Manager  2  reads  mail  at  5  p.m. 

0.1 

8.0 

19. 

Manager  2  drafts  answer  and  puts  it  into  the  outgoing  mailbox,  telling  manager  1  to  supply 
more  information.  It  is  classified  “rush”  because  it  is  now  overdue. 

0.3 

0.3 

TOTAL 

■  2.2  i;:\: 

170,4 

*  Source:  Harrington.  Dr.  H.J..  Business  Process  ImDrovement.  McGraw-Hill,  Inc.,  1991. 
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called  business  value-added  tasks.  The  ones 
that  show  the  highest  potential  for 
improvement  lie  in  transit  time,  waiting  time, 
review  time,  and  rework  time--they  have  no 
value-added  characteristics  from  the 
customer's  perspective. 

The  loss  of  time  is  one  perspective  of 
waste;  the  dollar  losses  may  be  even  more 
staggering.  Both  rework  and  review,  for 
example,  are  often  more  costly  than  the 
salary  losses  of  the  assigned  employee  who 
did  the  work  originally.  With  rework  and 
review  multiple  employees  get  involved; 
some  higher  graded  managers  and 
supervisors  will  add  to  the  costs  and  there  is 
time  taken  away  from  other  projects.  The 
multiplier  effect  ripples  through  the  system 
and  takes  it  toll. 

The  irony  of  this  situation  is  that  much 
of  this  is  being  charged  to  the  direct  labor 


account-the  mainstay  of  productivity  and 
efficiency  measurement.  Overhead  labor  has 
been  the  traditional  target  for  cost  control  but 
it  is  now  clear  it  is  not  the  only  culprit,  nor 
the  most  significant,  in  cost  control  efforts. 

Cycle  time  can  be  improved  and  waste 
reduced  often  because  they  may  be  simply 
quantified  and  exposed.  Exact  measures  are 
not  necessary  for  effective  cost  control.  In 
fact,  since  most  organizations  conduct  their 
business  by  stovepipe  or  division,  work 
consequently  flows  across  divisions  and 
offices  making  it  difficult  to  pinpoint  cycle 
time  problem  areas  exactly.  Any  exposure  or 
attention  given  to  these  issues  will  be  the 
beginning  of  change.  The  single  most 
important  theme  of  the  cycle  time  analysis  is 
that  quality  is  reduced  and  customer's 
expectations  go  unmet  when  waste  is  present. 
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B.l.  BENCHMARKING 


3  3  3  3  3 


Benchmarking  has  been  elegantly 
defined  as:  "a  continuous, 
systematic  process  for  evaluating 
the  products,  services,  and  work  processes  of 
organizations  that  are  recognized  as 
representing  the  best  practices  for  the 
purpose  of  organizational  improvement" 
[Spendolini,  1992],  Benchmarking  is  not  just 
the  establishment  of  a  performance  or 
numerical  target;  it  integrates  the  adoption 
and  understanding  of  the  process  which  made 
the  benchmarked  organization  superior  in  the 
first  place.  Benchmark  comparisons  may  be 
with  an  industry  (COE)  average,  best 
competitor,  world  class  performas,  or  best  in 
a  class.  Others  are  possible.  There  are  a 
number  of  ways  to  approach  benchmarking 
including:  internal  benchmarking, 

competitive,  functional,  and  statistical 
benchmarking.  These  will  be  discussed  in 
due  course. 

Many  benchmarking  approaches  are 
focused  or  directed  at  firms  which  are 
competing  in  the  market  place  for  private 
sector  profits  and  market  share.  Such  firms 
are  driven  by  the  profit  motive  in  order  to 
remain  liquid  and  vendable.  Such  notions  as 
market  price,  market  demand,  value  added, 
customer  et  al,  all  have  different  connotations 
in  the  private  sector  than  they  do  in  the 
public  sector  of  our  economy.  The  role  of 
government  is  to  fill-in  where  there  is  market 
failure.  In  the  context  of  public  institutions 
this  means  that  government  needs  to  step  in 
where  markets  do  not  exist  or  where  it  is  in 
the  collective  national  interest  to  provide  a 
product  or  service.  Classic  examples  are 


defense,  public  health,  police  and  fire 
protection,  and  pollution  abatement.  There 
are  no  "markets"  for  these  services.  The 
reason  this  issue  is  being  surfaced  is  that 
there  is  considerable  debate  as  to  whether  or 
not  the  government  should  benchmark 
against  the  private  sector.  The  methods  and 
motives  driving  the  public  and  private  sectors 
are  very  different;  why  should  they  be 
comparable?  Further,  the  laws  which  govern 
the  way  the  private  sector  operates  are 
different  from  those  under  which  public 
institutions  operate.  While  there  is  merit  to 
both  positions,  common  ground  is  possible. 
We  cannot  expect  government  institutions  to 
perform  in  the  same  manner  as  private  sector 
institutions,  but  we  can  still  benchmark 
against  the  private  sector  as  a  means  of 
calibrating  improvement.  In  other  words, 
while  we  may  not  expect  government  to  meet 
the  same  standards  as  the  private  sector,  we 
can  still  use  private  sector  performance 
results  as  a  reference.  In  the  context  of  this 
debate,  let  us  evaluate  the  merits  of  each  of 
the  benchmarking  types;  internal,  external, 
functional,  and  statistical. 

Internal  benchmarking  is  the  comparison 
of  one  department's  or  branch's  method  of 
doing  business  against  another's  in  the  same 
organization.  It  is  assumed  that  departments 
or  branches  operate  and  exist  in  different 
locations  but  the  goal  here  is  to  seek-out 
business  practices  that  are  better  in  one 
department  than  anywhere  else  and  adopt 
those  practices  throughout  the  corporate 
structure.  For  the  COE  this  would  require 
that  districts  and  divisions  exchange 
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information  so  that  comparisons  can  be 
made.  As  is  apparent,  this  must  take  place 
between  departments  and  branches  serving 
the  same  functions  i.e.  planning  with 
planning,  engineering  with  engineering,  in 
different  districts  although”best  practices” 
can  be  taken  from  different,  unrelated 
organizations.  The  best  way  to  do  this  is  to 
establish  common  data  and  performance 
measures  or  indexes  with  at  least  several 
benchmark  "partners".  This  is  a  form  of 
competition  but  is  driven  from  within  rather 
than  externally.  It  also  fosters 
communications.  The  best  internal  practices 
are  discovered  and  transferred  to  other 
operations.  Table  lb  provides  an  example  of 
four  districts  exchanging  what  they  consider 
to  be  key  organizational  cost  drivers.  It  is 
apparent  from  the  data  exchanged  that  some 
districts  have  something  to  learn  from  others. 
District  "B"  has  something  to  gain  from  the 
others  in  the  area  of  G&A  costs  (General  and 
Administrative  costs  which  includes 
personnel,  legal,  procurement,  supply, 
administrative,  office  rental,  equipment, 
etc.),  district  "C"  has  something  to  learn 
about  annual  equipment  repair  costs,  and 
district  "D"  needs  to  look  at  what  is  driving 
direct  labor.  This  exchange  of  information 
can  lead  to  gains  by  all  involved.  It  is 
possible  to  expand  "partner"  membership 
between  other  cities  and  districts  by  using 
price  deflators  to  adjust  for  differences  in  the 
cost  of  living  between  cities.  Such  ready¬ 
made  indexes  are  available  on  a  monthly 
basis. 

Competitive  benchmarking  requires  that 
each  element  compare  its  business  processes, 
products,  and  services  with  those  of  a 
competitor.  This  type  of  benchmarking 
alludes  to  a  problem  previously  discussed; 


many  government  activities  are  public  goods 
and  there  are  no  real  market  competitors  for 
such  missions.  However,  contractors  can  be 
used  in  many  cases  as  the  benchmark  for 
those  activities  that  we  have  contracted  in  the 
past.  The  difficulty  with  this  approach  is  in 
obtaining  data.  Most  firms  are  not  too 
willing  to  release  information  on  the  cost  of 
doing  business;  there  is  a  fear  that  such 
information  will  wind  up  in  the  hands  of 
those  with  whom  they  themselves  must 
compete.  While  some  data  are  available  in 
published  sources  which  may  be  of  value,  it 
is  usually  too  gross  and  non-specific  to  be  of 
real  use.  There  are  organizations  which 
collect  and  publish  this  kind  of  information 
as  their  business  but  once  again,  water 
resource  organizations  are  difficult  to  come 
by  and  comparing  yourself  with  "cousin" 
resource  firms  who  produce  similar  outputs 
may  be  misleading.  It  may  be  possible  to 
arrange  agreements  not  to  publish  each  others 
cost  of  doing  business  or  business  processes 
and  practices  but  this  may  be  limited  in  scope 
and  a  little  tenuous.  While  the  competitive 
approach  is,  in  some  sense,  the  best  and  most 
direct  method  of  benchmarking,  it  is  also  the 
most  difficult  to  contend  with.  Table  lb 
illustrated  earlier  provides  an  example  of  the 
type  of  comparisons  which  could  be  made  if 
competitive  benchmarking  were  instituted. 

Functional  or  generic  benchmarking  is 
another  approach.  This  technique  involves 
identifying  and  comparing  products  and 
services  of  organizations  that  are  not  direct 
competitors.  That  is,  find  the  best  in  the 
business  regardless  of  similarity  to  yours  and 
determine  what  can  be  adopted  to  fit  your 
needs.  This  can  be  done  for  specific  business 
practices  such  as  cycle  times,  quality 
attributes,  design  practices,  planning 
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techniques,  etc.  The  real  focus  is  on 
business  processes  rather  than  on  specific 
services  or  products;  but,  all  activities  are 
fair  game.  Like  the  previous  benchmarking 
measures,  data  may  be  difficult  to  obtain. 
Once  again,  Table  lb  could  be  the 
comparative  format  used. 

Table  lb  can  employ,  not  only  univariate 
data,  but  also  single  and  multifactor  index 
numbers,  efficiency  measures,  and  other 
performance  measures.  The  table  could  be 
structured  by  function,  KRA,  program  or 
another  desired  format. 

Statistical  benchmarking  may  be  the 
most  viable  approach.  It  involves  securing 
key  data  from  each  and  every  district  or 
division  in  the  COE  and  constructing  a 
distribution  of  performance  activities.  All 
COE  elements  would  have  to  participate  in 
order  to  establish  a  large  enough  sample  size 
for  statistical  estimation  and  ranking.  This 
would  be  a  very  powerful  tool,  fostering 
much  competition.  The  data  would  be 
relatively  easy  to  obtain  since  each  district 
and  division  will  be  on  a  performance 
measurement  program  in  the  near  future.  We 
would  have  to  be  certain  that  all  elements  are 
using  the  same  accounting  systems  and 
practices  so  that  we  do  not  compare  apples 
with  oranges.  Such  a  technique  would  allow 
for  comparisons  against  the  organizational 
mean,  establish  modes  and  medians, 
percentile  ranks,  determine  variances  and 
standard  deviations,  and  establish  reliability 
of  results.  This  could  be  done  for  each  key 
parameter  or  KRA,  etc.  To  a  certain  degree 
it  would  also  allow  for  comparisons  with 
private  sector  practices  when  such  data 
became  available.  Tables  2b,  3b,  and  Figure 
lb  will  serve  as  an  illustration  of  this 
approach. 


Table  2b  shows  a  list  of  hypothetical 
data  used  to  illustrate  the  use  of  statistical 
benchmarking.  Column  (1)  represents  a 
numerical  value  for  each  district  in  the  COE. 
Column  (2)  is  the  General  and  Administrative 
(G&A)  cost  for  each  district.  Columns  (3), 
(4),  and  (5)  are  the  frequency  of  observing 
each  G&A  cost,  the  percentage  occurrence  of 
each  G&A  cost,  and  the  cumulative 
percentage  of  districts  in  the  database, 
respectively.  G&A  costs  are  expressed  in 
dollars  per  FTE  and  are  found  by  dividing 
the  sum  total  of  district  G&A  expenditures  by 
the  total  number  of  full  time  equivalent 
employees  for  each  district. 

Table  3b  takes  the  data  in  Table  2b  and 
summarizes  it  by  assigning  each  G&A  value 
to  a  class  interval  i.e.  how  many  districts 
show  a  G&A  cost  of  between  $15,000  and 
$15,999,  how  many  show  a  G&A  cost  of 
between  $16,000  and  $16,999,  etc.  The  data 
have  been  adjusted  for  the  cost  of  living  in 
each  city  in  which  each  district  is  located  to 
prevent  distortion  of  information.  These 
dollar  intervals  are  called  class  intervals  in 
statistics.  The  count  column  are  the  number 
of  districts  observed  in  each  class  interval  . 
The  percent  column  shows  the  percentage  of 
values  in  the  count  column.  Notice  the  data 
following  the  table;  various  statistical 
characteristics  are  shown. 

Figure  lb  is  the  data  from  Table  3b 
plotted  in  the  traditional  frequency 
distribution  format.  The  data  are  nearly 
normally  distributed  judging  by  the  shape  of 
the  distribution  and  the  proximity  of  the 
mean  and  median.  The  curve  depicts  a  good 
deal  of  information  which  is  ideally  suited  for 
a  taxonomic  arrangement  of  district 
expenditures  on  G&A.  For  example,  those 
districts  falling  in  the  class  interval  between 
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TABLE  lb 

BENCHMARK  INDICATORS 
EXAMPLE:  AN  ENGINEERING  ORGANIZATION 


1  All  costs  adjusted  for  city  of  location. 
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Median  =  $19,000 
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Figure  lb.  -  Frequency  Distribution  of  General  and  Administrative  Costs  (G&A) 


TABLE  3b 

DISTRIBUTION  OF  PERFORMANCE  1 


RANGE 

COUNT 

PERCENTAGE  %  | 

$15,000  -  15,999 

2 

5.6  ! 

16,000  -  16,999 

3 

8.3  || 

17,000  -  17,999 

5 

i3.9  | 

18,000  -  18,999 

7 

19.4  || 

19,000  -  19,999 

9 

25.0  || 

20,000  -  20,999 

6 

16.7  || 

21,000-21,999 

3 

8.3  j  I 

22,000  -  22,999 

1 

2.8  ]: 

36 

100.0 

1  All  data  price  adjusted  for  city  of  location. 


Mean  =  $18,872 
Mode  =  N.A. 

Median  =  $19,000 
Std  Dev  =  $1,791 
Sample  Size  =  36 
Range  =  $15,000  -  22,600 
N.  A.  =  Multiple  Modes 

$15,00  and  $15,999  per  FTE  fall  in  the 
lowest  5.6%  of  all  districts  included  in  the 
sample  count.  In  this  case  two  districts  fit 
this  description.  Thus  their  G&A  costs  are 
among  the  lowest  94.4%  in  the  COE.  The 
next  group-with  a  class  interval  of  G&A 
costs  between  $16,000  and  $16, 999- 
represent  8.3%  of  the  districts  and  fall  within 
the  lowest  13.9%  of  all  districts  or; 
alternatively  their  G&A  costs  are  lower  than 
76.1%  of  all  districts.  For  comparison, 
those  districts  which  fall  into  the  class 
interval  with  G&A  costs  per  FTE  ranging 


from  $22,000  to  $22,999  represent  2.8%  of 
all  districts  and  are  in  the  upper  97.2%  of  all 
districts  in  terms  of  G&A  costs.  Clearly, 
those  districts  in  the  highest  class  intervals 
have  something  to  learn  from  those  in  the 
lower  G&A  cost  areas. 

The  point  in  all  this  is  that  internal 
benchmarking  is  possible  with  the  statistical 
approach  and  measurements  among  and 
between  districts  can  be  made  so  that 
business  processes  can  be  compared.  Notice 
also  that  other  statistics  are  provided  such  as 
the  mean,  median,  and  the  standard 
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deviation.  This  will  allow  for  the  ranking  of 
districts  in  terms  of  performance  for  each 
KRA  or  attribute  measured.  For  example,  it 
is  known  that  68.3%  of  all  the  data  points 
fall  between  one  standard  deviation  of  the 
mean.  This  means  that  66.3%  of  the  districts 
fall  within  the  range  of  $17,091  and  $20,663; 
this  is  the  norm  of  the  data.  Similarly, 
statistical  information  states  that  95.5%  of 
the  data  must  fall  between  two  standard 
deviations  of  the  mean.  This  carries  a  range 
of  $15,290  and  $22,454.  Those  districts 
falling  outside  this  range  represent  the  very 
best  performers  (below  $15,290  in  G&A 
costs)  and  worst  performers  (above  $22,454). 
The  value  of  the  statistical  approach  to 
benchmarking  is  very  powerful. 

External  benchmarking  is  also  possible 
using  the  statistical  approach.  If  we  know 
that  comparable  private  sector  firms  typically 
show  a  G&A  cost  per  FTE  of  $17,500,  then 
we  can  compare  this  with  the  COE  frequency 
distribution.  In  this  case  the  private  sector 
organizations  fall  within  the  class  interval  of 
$17,000  to  $17,999.  This  means  that  they 
are  better  than  average  performers  in  G&A 
expenditures  relative  to  the  COE.  They 
would  fall  approximately  in  the  lower  17  %  of 
the  cost  curve. 

Normalized  Data 

When  employing  benchmarking  for 
comparative  purposes,  be  certain  your  data  is 
compatible  with  that  of  the  benchmark’s.  It 
of  little  value  to  use  information  which  is 
similar  in  name  but  quite  different  in  actual 
content  and  meaning.  For  example,  cost  per 


employee  is  a  common  marker  for 
comparison.  If  you  are  using  direct  cost  only 
and  the  benchmark  organization  is  including 
fringe  benefits  as  part  of  the  cost  of  labor, 
the  data  are  not  comparable  and  would  lead 
to  incorrect  conclusions  and  inappropriate 
goals  and  objectives.  This  difference  would 
not  only  impact  the  level  of  the  numbers,  but 
the  rates  of  change  over  time  as  well.  This  is 
not  an  isolated  circumstance.  Even  though 
we  all  work  for  the  same  organization, 
accounting  practices  are  not  one  hundred 
percent  alike.  They  may  differ  by  local 
custom  or  tradition  and  be  taken  for  granted 
without  the  differences  being  readily 
apparent.  Overhead  in  one  district  may  be 
28  percent,  while  in  another,  within  the  same 
division,  it  may  be  30  percent;  the  difference 
may  not  be  in  operational  practices  but  in 
accounting  practices.  These  types  of  data 
compatibility  problems  can  be  found  in  non- 
financial  data  as  well.  In  benchmarking  or  in 
competing  for  the  ACOE  award,  this 
difference  may  be  a  deciding  factor.  The 
expense  is  not  universal.  The  situation  may 
be  compounded  when  attempting  to 
benchmark  with  another  agency  or 
organization  because  institutions  which  are 
non-governmental  may  even  have  altogether 
different  definitions  for  the  same  cost 
category.  The  lesson  is  to  be  certain  that 
when  benchmark  data  are  used  as  “stretch 
targets”,  goals  or  objectives,  have  local 
experts  talk  with  one  another  to  be  certain 
that  the  data  being  used  or  “group  shared” 
are  congruous.  This  will  save  a  lot  of 
headaches  later. 
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B.2.  MEASURES  OF 
FLEXIBILITY 
AND  AD AP ABILITY 
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One  of  the  central  credos  of  a 
high  performance  organization 
is  its  ability  to  adapt  to  change. 
This  is  more  true  today  than  ever  before 
because  of  the  rapid  change  in  technology 
and  the  shift  toward  customer  satisfaction. 
Customers  are  not  only  demanding  a  higher 
quality  product  or  service  at  a  lower  cost,  but 
are  demanding  much  faster  cycle  times. 
Improved  cycle  time  may  be  the 
quintessential  ingredient  to  an  organization's 
survival  in  today's  consumer  environment. 
Adaptability  and  flexibility  are  difficult  to 
measure  and  little  research  is  available  on  the 
subject.  This  part  of  the  text  will  describe 
adaptability  and  flexibility  and  provide 
several  measures  which  can  serve  as  proxies 
for  their  measurement.  To  simplify  matters 
flexibility  and  adaptability  will  be  viewed  as 
the  same  notion.  Adaptability  can  be 
described  as  the  ability  of  an  organization  to 
adjust  or  to  accommodate  new  environments 
through  the  use  of  skill,  knowledge  or 
judgement.  There  is  also  a  connection  with 
benchmarking.  Prior  to  delving  into  the 
measures  of  adaptability  a  few  additional 
concepts  need  to  be  introduced. 

Service  or  knowledge  work  usually 
consists  of  three  categories  of  projects.  They 
include  recurring  projects,  long-term 
projects,  and  quick  response  projects  [Boyett 
and  Conn,  1988]. 

Recurring  work  is  characterized  by  the 
fact  that  it  is  regularly  scheduled,  usually 
involves  a  standard  procedure,  and  has  a 
standard  due  date  or  expected  time-frame  for 
completion  [Boyett  and  Conn,  1988]. 


Typical  examples  would  include  periodic 
manpower  reports,  accounting  or  budgeting 
reports,  status  reports,  et  al.  Such  reports  are 
expected  to  be  completed  on  a  weekly, 
monthly,  or  quarterly  basis. 

Long  term  projects  extend  over  several 
months  or  years,  usually  have  a  schedule, 
process,  or  budget,  and  are  usually  one  time 
projects  [Boyett  and  Conn,  1988].  They 
have  a  standard  or  particular  procedure 
unique  to  themselves  even  though  the  general 
process  may  be  repetitious. 

Quick  response  projects  are  the  ones 
which  are  the  most  susceptible  to  measures  of 
adaptability.  They  are  generally  not 
repetitive,  typically  demand  a  speedy 
response  time  (hours  or  days)  and  often  do 
not  have  a  standard  procedure  [Boyett  and 
Conn,  1988].  Corps  of  Engineering 
personnel  often  refer  to  these  projects  as 
"flash  fires".  They  are  disruptive  to  on¬ 
going,  long  term  activities  such  as  those 
described  earlier.  Within  the  context  of  cycle 
time,  quick  response  projects  may  even  be 
described  as  "waste"  or  non- value-added 
activity  because  they  require  employees  to 
drop  what  they  are  doing  and  respond  to  the 
"fire".  Quick  response  projects  are  most 
often  not  scheduled  and  typify  a  hospital 
emergency  room  mentality.  Nearly  every 
organization  has  to  deal  with  them.  There  are 
no  known  direct  measures  of  quick  response 
for  these  type  of  work  activities.  Proxies 
are,  however,  useful. 

There  are  several  indirect  measures  of 
organizational  adaptability.  They  include: 
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□  The  average  completion  time  per  quick 
response  request, 

□  The  ratio  of  the  number  of  different 
disciplines  available  to  an 
organization's  total  staff,  times  one 
hundred, 

□  The  average  number  of  quick 
response  requests  completed  per  unit 
of  time, 

□  The  ratio  of  bonafide  skills  available 
to  the  organization's  total  staff,  times 
one  hundred, 

□  The  cumulative  number  of  hours  of 

multidisciplinary  formal  training 

completed. 

Each  of  these  will  be  described  in  turn. 

The  average  completion  time  per  quick 
response  request  measures  the  speed  with 
which  requests  are  handled  relative  to  some 
over-all  benchmark  such  as  the  mean 
completion  time  for  previous  quick  response 
requests.  This  is  similar  to  the  application  of 
the  frequency  distribution  described  earlier. 
By  comparing  the  historic  average 
completion  time  to  current  year  response 
time  one  is  able  to  determine  if  response  time 
is  improving  relative  to  a  base  period  of 
historic  average  completion  times.  This  is 
purely  a  measure  of  time  to  complete  a 
request.  Figure  2b  provides  an  illustration. 
There  are  two  frequency  distributions  in  the 
figure.  One  representing  an  historic  time 
response  to  special  requests  with  a  mean 
response  time  of  30  hours.  This  distribution 
is  the  culmination  of  say  the  response  times 
from  last  year  or  the  accumulated  response 
times  from  several  previous  years.  The 
second  distribution  shows  a  mean  of  10  hours 
and  represents  the  results  of  a  current  year 
record.  Note  that  the  mean  response  time 


has  dramatically  shifted-improved-  over  time 
from  distribution  one  to  distribution  two. 
This  is  the  kind  of  measurement  needed  to 
document  improvements. 

The  ratio  of  the  number  of  different 
disciplines  available  to  an  organization's  total 
staff  multiplied  by  one  hundred  is  the  percent 
of  staff  with  different  backgrounds;  an 
indicator  of  multidisciplinary  capability. 
This  is  a  proxy  for  employee  multi  skill 
capability  to  respond  to  varying 
circumstances.  The  time  it  takes  to  respond  is 
directly  proportional  to  the  technical 
capability  to  respond.  A  frequency 
distribution  such  as  the  one  just  described  in 
Figure  2b  could  also  be  constructed  to 
measure  and  display  improvement  over  time. 

A  third  measure  is  the  average  number  of 
quick  response  requests  completed  per  unit  of 
time.  This  is  another  speed  of  response 
measure.  Again  the  average  is  used  because 
an  individual  response  is  almost  meaningless 
to  the  measurement  of  improvement  in 
response  time  over  a  historic  pattern  and  as 
before,  a  statistical  benchmark  can  be 
established.  Another  frequency  distribution 
can  be  used  here  as  well. 

The  ratio  of  bonafide  skills  available  to 
total  number  of  staff  times  one  hundred  is 
another  capability  measure.  It  is  a  measure  of 
the  percent  of  staff  with  multidisciplinary 
capability.  This  measure  may  include 
licensed  individuals,  sanctioned  and 
documented  cross-training  in  different  skill 
areas,  or  some  combined  measures  of  both  of 
these.  The  diversity  of  cross-training  is  more 
important  than  the  degree  of  specialization  in 
a  particular  skill  in  measuring  response  time 
to  special  requests.  Once  again,  a  tabular 
value  an  be  used  or  a  frequency  distribution 
van  be  employed. 
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RESPONSE  TIME  IN  HOURS 


Figure  2b.  -  Measures  of  Adaptability  and  Flexibility 
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The  cumulative  number  of  hours  of  formal 
multidisciplinary  training  completed  is  also  a 
measure  of  response  capability.  The  more 
multidisciplinary  training  individuals  have, 
the  more  they  are  capable  of  responding  to 
requests  outside  their  principle  discipline. 
Cumulative  hours  is  a  good  proxy  for 
cumulative  capability.  If  this  training  has  not 
been  used  for  a  specified  time-say  two  years- 
then  it  may  no  longer  be  applicable  to  the 
cumulative  total  for  the  organization.  Unused 
skills  and  training  diminish  over  time.  Thus 
a  running  cumulative  total  (cumulative  total 
less  time  depreciated  hours  not  utilized)  is  a 
more  accurate  indicator  of  real  capability.  An 
adjustment  must  also  be  made  for  gains  and 
losses  in  FTE's.  Note  the  connection  between 
adaptability  and  the  concept  of  knowledge 
discussed  earlier.  This  nexus  is  not  passive 
nor  accidental  and  signifies  the  importance  of 
knowledge  and  training  once  again. 

Although  it  is  beyond  the  scope  of  this  text 
to  describe,  the  frequency  distributions 
discussed  above  provide  more  information 
than  indicated.  Using  "Z"  scores,  skew 


analysis,  risk  concepts,  and  various  kinds  of 
test,  more  information  is  obtainable  than 
described  here.  There  are  other  measures  of 
adaptability  which  may  be  useful  in 
determining  an  organization's  response  to 
changing  circumstances.  For  example,  the 
number  of  changes  in  core  management 
techniques  put  in  place  by  management  could 
be  another  measure  of  an  ability  to  adjust. 
This  would  include  many  of  the  principles 
and  practices  covered  in  this  text  (continuous 
process  improvement,  SPC,  performance 
measurement,  and  GPRA  ideas  made 
operational).  From  a  technical  viewpoint, 
the  cumulative  and  absolute  number  of  new 
tools  and  techniques  used  is  another  proxy 
for  measuring  adaptability. 

The  ability  to  accept  change  and  engender 
new  ways  is  a  measure  of  survivability.  The 
world  has  been  in  a  new  phase  of  cultural 
change  since  the  1970's  which  may  be  more 
significant  than  the  Industrial  Revolution  of 
the  past  two  centuries.  Those  not  able  to 
adapt  will  not  survive. 
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PROCESS  AND  SYSTEM  PERFORMANCE  MEASUREMENT 
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C.l.  SYSTEM  PERFORMANCE 
MEASUREMENT  AND 
PROCESS  MAPPING 


The  Government  Performance  And 
Results  Act  of  1993  (GPRA)  and 
the  National  Performance  Review 
(NPR)  and  the  APIC  criteria  mentioned 
earlier  all  emphasize  the  development  of 
goals  and  objectives  and  measurement  of 
organizational  performance  in  achieving 
those  goals.  (Generally,  these  are  increased 
efficiency,  effectiveness,  and  outcome  with 
considerable  weight  given  to  customer  needs 
and  wants.)  In  this  section  of  the  hand  book, 
you  will  be  given  simple  but  effective  tools 
and  methods  for  measuring  system  and 
process  performance.  You  will  find  that 
measurement  usually  focuses  on  those 
activities  we  define  as  “core”  processes. 
These  are  the  processes  which  are  critical  to 
the  mission.  They  are  either  elements  of  the 
mission  itself,  or  support  activities  which 
provide  an  indirect,  but  crucial,  complement 
to  the  mission.  Obviously,  improvements  in 
these  processes  can  be  expected  to 
significantly  affect  productivity. 

This  part  of  the  report  will  be  used  for 
exploring  performance  problems  in  complex 
management  and  technical  systems  and 
processes  such  as  those  found  in  the  Army 
Corps  of  Engineers.  A  discussion  of  systems 
and  processes  was  presented  earlier  and  the 
reader  may  wish  to  review  that  section  prior 
to  reading  the  following  paragraphs.  This 
section  will  also  show  how  benefits  can  be 
evaluated  using  network  analysis  (process 
mapping)  and  how  to  employ  a  benefit-to-cost 
ratio  (BCR)  to  determine  whether  or  not  an 
alternative  solution  to  a  performance  problem 
or  system  malfunction  should  be  undertaken. 


Or,  which  of  a  number  of  plausible  solutions 
should  be  undertaken.  The  principle  criteria 
are  three:  customer  satisfaction, 

effectiveness,  and  efficiency. 

Systems  and  Process  Measurement 

Two  of  the  essential  ingredients  necessary 
for  gains  in  performance  are  the  application 
to  processes  and  systems  and  the  notions  of 
continuous  improvement  and  problem 
prevention.  Process  mapping  is  the 
important  tool  for  this  effort.  Continuous 
improvement  is  the  persistent  day-to-day 
search  for  all  organizational  performance 
problems  and  the  implementation  of  their 
solutions.  This  effort  must  embody  the 
entire  organization  (employees  and 
management)  and  be  applied  to  all  problems- 
large  and  small.  It  is  this  constant 
incremental  effort  (the  concept  of  kaizen  in 
Japanese)  of  finding  even  the  smallest 
problem  and  its  solution  that,  over  time,  adds 
up  to  significant  improvements  in 
effectiveness  and  efficiency.  The  reader 
should  be  aware  that  sudden  and  dramatic 
improvements  (the  concept  of  hoshin  or 
break-throughs  in  Japanese)  are  part  of  the 
process  and  will  occur  along  with 
incremental  changes.  They  will  likely  be 
spontaneous  or  serendipitous  rather  than  a 
planned  part  of  change.  All  employees  must 
be  involved  because  it  is  they,  each  at  their 
own  tasks,  that  know  where  the  problems 
are.  Business  process  mapping  and 
organizational  problem  solving  can  be  done 
from  the  bottom  up  contrary  to  the  classical 
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model  of  hierarchy,  previously  discussed. 
Management  does  not  have  the  tacit 
knowledge  that  employees  do.  The  effort  to 
seek  out  and  discover  why  something  is  not 
satisfying  customers,  needs  to  be  continuous 
and  not  a  one-time  undertaking  as  is  often  the 
case  in  most  organizations.  This  collective, 
continuous  action  leads  to  the  reduction  of 
waste,  rework,  and  redundancy  resulting  in 
decreased  cost  of  products  and  services  and 
improved  quality.  This  is  the  embodiment  of 
continuous  improvement  and  constancy  of 
purpose. 

The  advantages  of  quantifying 
organizational  operations  are  very 
convincing.  The  numbers  tell  us  how  much 


change  is  taking  place  as  a  result  of  some 
corrective  action  and  in  what  direction  the 
performance  is  going.  Another  quality  of 
measurement  lies  in  benchmarking  the  results 
against  the  "best  in  the  business"  as 
previously  discussed.  Publication  of  the 
performance  numbers  serves  as  a  motivating 
force  for  the  organization.  If  performance 
improves  over  time,  then  this  alone  serves  to 
encourage  participation  and  provides  a  sense 
of  ownership  in  the  solution  among 
employees.  If  performance  decreases,  this 
can  serve  to  motivate  as  well  by  letting  the 
organization  know  that  its  current  effort  or 
improvement  program  needs  to  change 
direction-feedback  is  vital. 
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A  network  methodology  such  as  the 
Program  Evaluation  And  Review 
Technique  (PERT)  is  a  tool 
originally  developed  to  organize  and  monitor 
one-time,  complex  undertakings  such  a 
missile  programs,  housing  projects,  and  large 
scale  water  resource  programs  and  projects. 
It  is  easily  modified  and  adapted  for 
application  to  examining  and  monitoring 
organizational  operations  and  the  impacts  of 
organizational  structural  changes  and 
business  operations.  It  fits  the  model  of 
continuous  improvement  and  constancy  of 
purpose  discussed  above.  It  also  is  very 
functional  in  incorporating  customer  needs 
and  wants  and  in  evaluating  improvements  in 
product  and  service  effectiveness  and 
efficiency. 

Prior  to  examining  an  application  of  the 
PERT  methodology  and  how  it  applies  to 
developing  the  benefits  and  costs  of 
performance  improvements,  it  is  necessary  to 
briefly  discuss  what  it  does  and  how.  Figure 
lc  shows  a  simple  operational  flow  of  part  of 
an  organization.  Let  us  start  with  some 
hypothetical  process  designed  to  transform 
resources  into  a  product  or  service  for 
customers.  A  goal  is  a  broadly  stated 
refinement  of  mission  concerning  key 
organizational  issues.  The  goal  in  this  case 
is  to  reduce  operating  costs.  An  objective  as 
stated  earlier  is  a  specific  statement  of  a  goal 
elaborating  the  results  anticipated.  The 
objective  in  this  case  is  to  reduce  plan 
formulation  process  costs  by  at  least  5  %  over 


the  next  five  years  without  adversely 
impacting  customer  satisfaction  (quality).  It 
starts  with  "A"  representing  the  initial  event 
"A-B".  The  process  of  going  from  "A-B" 
requires  an  organization  to  expend  resources 
to  accomplish  some  activity  or  transform 
resources  into  products  or  services.  These 
resources  usually  include  management, 
money,  employee  labor,  land,  capital, 
technology,  knowledge,  etc.  Once  activity 
"A-B"  is  accomplished,  the  incomplete 
product  or  service  must  undergo  additional 
transformation  (move  from  one  department 
or  branch  to  another)  as  in  this  case  of 
moving  from  "B"  to  "C"  and  from  "B"  to 
"F"  and  from  "B"  to  "E".  These  three  new 
activities  further  transform  the  product  or 
service  and  add  value  as  the  product  or 
service  moves  toward  completion.  The 
arrows  represent  direction  of  flow  and  the 
time  it  takes  to  go  from  event  "A"  to  event 
"B"  or  from  event  "B"  to  event  "C".  Each 
event  pair  represents  an  activity  which  is  to 
be  accomplished.  The  number  above  or  to 
the  side  of  the  arrow  represents  the  time 
(i.e.,  8  days,  8  weeks,  8  months,  etc.)  it 
takes  between  events  and  associated 
activities.  The  entire  network  in  this  simple 
system  goes  from  event  "A"  to  event  "H"  but 
passes  through  transformation  processes 
"A-B,B-C,B-E,B-F,D-G,F-G  and  G-H" 
(from  one  office,  branch,  or  division  to 
another).  This  is  the  essence  of  the  PERT 
process  model.  It  is  a  "map"  of  the  business 
process,  what  is  accomplished,  how  it  is 
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Figure  lc.  -  Existing  Business  Process  Map 


accomplished,  the  time  it  takes  to 
accomplish,  and  if  plotted  as  such,  the 
sequence  of  events  in  some  parallel  or  series 
fashion.  The  tie-in  with  customer 
satisfaction  can  be  specified  by  letting  event 
"A"  represent  a  list  of  customer  needs, 
wants,  or  specifications  and  event  "H" 
represents  the  comparison  of  customer 
specifications  with  the  final  product  or 
service.  If  the  customer  is  not  satisfied,  it 
will  be  necessary  to  use  this  feedback  to  trace 
through  the  process  to  see  where  things  went 
wrong  and  make  appropriate  adjustments. 

One  final  point  needs  to  be  addressed 
concerning  PERT  analysis.  The  path  of  the 
longest  time  in  the  network  or  process  is 
called  the  "critical  path".  Table  lc  shows  the 
various  paths  through  the  process  or  system 
with  path  "A-B-F-G-H"  being  the  critical 
path.  The  user  needs  to  compare  this  with 
some  preestablished  schedule  to  determine  if 
the  timing  in  the  PERT  can  meet  the 
specified  deadline.  PERT  can  be  used  not 
only  to  evaluate  the  steps  and  order  in  a 
process  but  can  be  employed  to  schedule 
completion  times-another  customer  quality 
factor. 

Table  2c  shows  some  hypothetical  data 
which  will  be  adopted  to  illustrate  how  PERT 
can  be  used  to  map  out  and  trace  through  a 
process  which  may  represent  a  subsystem  of 
a  large  organization.  It  is  tied  to  Figure  lc 
discussed  earlier.  Column  (1)  of  Table  2c 
represents  the  events  and  activities  associated 
with  Figure  lc.  Column  (2)  shows  the 
expected  completion  times  for  each  of  the 
activities.  Column  (3), (4), (5)  and  (6)  show 
the  number  of  personnel  required  to  complete 
each  activity,  the  hours  per  day  of  effort  per 
person,  the  dollars  per  hour  paid  each 
employee  (direct  labor,  indirect  labor, 


overhead,  etc.)  and  the  number  of  days 
worked  per  week,  respectively.  Column  (7) 
represents  the  total  cost  of  each  activity.  In 
this  simplified  illustration  other  resource  such 
as  capital,  technology,  raw  materials,  etc., 
are  left  out  to  avoid  complicating  the  example 
but  in  reality  all  resource  costs  need  to  be 
included.  Note  that  the  total  cost  of 
completing  the  needed  services  or  product  for 
the  customer  are  $517,800. 

Suppose  that  a  re-analysis  of  the  process  in 
Figure  lc  demonstrates  that  event  "C"  can  be 
combined  with  event  "D"  to  reduce  the 
number  of  steps  in  the  process  and 
consequently  reduce  service  or  production 
costs.  This  may  be  possible  due  to  the 
application  of  some  new  management 
technique  which  can  enhance  the  productivity 
of  the  B-C  activity  thereby  reducing 
operating  costs.  It  could  also  stem  from 
uncovering  waste,  redundancy,  and  excess 
review  in  the  B-C  activity.  There  are  a  host 
of  tools  which  can  be  employed  to  explore 
and  detect  such  problems  including  Shewhart 
control  charts,  Pareto  charts,  and  cause-and- 
effect  charts  just  to  name  a  few.  The  cause 
not-with-standing,  the  new  process  illustrated 
in  Figure  2c  now  yields  a  new  operational 
cost  as  shown  in  Table  3c.  Note  that  the 
events  B-C,  C-E,  and  C-D  have  been 
consolidated  into  activity  B-D  as  part  of  the 
restructuring.  As  you  trace  through  the  costs 
and  the  events  of  restructuring,  you  will  note 
that  the  new  total  costs  for  the  same  service 
or  products  is  now  $460,800  which 
represents  a  savings  of  $66,000.  The  net 
result  is  an  12.7%  cost  reduction.  In  this 
particular  case  there  is  no  improvement  in 
delivery  time  because  the  critical  path  is  still 
the  same  (36  weeks)  as  before  but  total  costs 
are  none-the-less  reduced. 
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TABLE  lc 
PERT  TIME  PATHS 


Events  &  Activities 


A-B-E-G-D 


A-B-C-E-G-H 


A-B-C-D-G-H 


A-B-F-G-H 


TABLE  2c 

RESTRUCTURED  PROCESS  COST  BY  ACTIVITY 


COMPLETION 


NO.  OF 


ACTIVITY  TIME  (WKS)  PERSONNEL  HOURS/DAY  PER  HR 


DOLLARS  EFFORT  (# 


OF  DAYS) 


TOTAL 

COST 


$  96,000 
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Figure  2c.  -  Improved  Business  Process  Map 


TABLE  3c 

RESTRUCTURED  PROCESS  COST  BY  ACTIVITY 


1  1 

II  (1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7)  | 

||  ACTIVITY 

COMPLETION 
TIME  (WKS) 

NO.  OF 
PERSONNEL 

HOURS/DAY 

DOLLARS 
PER  HR 

EFFORT  (# 
OF  DAYS) 

TOTAL  If 
COST 

ij: ;!;  A-B 

8 

4 

8 

$75 

5 

$96,000  || 

11  B-C2 

6 

1 

8 

$75 

5 

0  || 

(I  B-E 

4 

7 

8 

$115 

5 

$128,800  || 

11  E"G 

3 

1 

8 

$25 

5 

$3,000  || 

||  C-E' 

6 

1 

8 

$65 

5 

0  1 

II  C-D' 

2 

8 

8 

$90 

5 

i-i  | 

0  || 

||  D-G 

10 

4 

8 

$20 

5 

$32,000  || 

II  GH 

6 

6 

8 

$  15 

5 

$21,600  || 

II  B“F 

12 

4 

8 

$60 

5 

$115,200  || 

|1  F-G 

10 

1 

8 

$90 

5 

$36,000  | 

|1  B-D2 

12 

2 

8 

$20 

5 

$  19,200  || 

TOTAL 

= 

$451,800  || 

($517,800  -  $45 1 ,800)/$5 17,800  =  12.7% 


1  Processes  Removed 

2  Processes  Added 
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The  savings  or  benefit  analysis  requires 
several  additional  steps  in  order  to  show  the 
full  impact  of  the  management  change.  The 
consolidation  is  expected  to  have  a  savings 
impact  for  at  least  five  years  into  the  future. 
Table  4c  shows  what  this  entails.  Column 
(1)  shows  known  future  studies  that  will 
benefit  from  the  new  operating  procedure. 
Column  (2)  shows  the  expected  savings  for 
each  of  five  forthcoming  studies.  Columns 
(3)  and  (4)  show  the  discounting  effect  used 
to  take  into  consideration  the  time  value  of 
money.  The  sum  of  the  discounted  values  is 
amortized  over  five  years  at  8-1/2%.  If  the 
average  annual  savings  is  expected  to  provide 
for  longer  term  gains  than  a  different 
amortization  period  could  be  used  and  vice 
versa.  The  final  factor  to  consider  is  the  cost 
of  implementing  the  new  operating 
procedure. 

Table  5c  shows  estimated  costs  for  putting 
the  changes  into  effect.  Costs  should  include 
all  resources  associated  with  the  action.  In 
this  simplified  example  a  few  likely  costs  are 
shown  to  illustrate  the  concepts  involved. 
Note  that  costs  must  undergo  amortization  as 
did  benefits  or  savings.  Costs  are  expected 
to  be  incurred  as  a  one  time  expenditure  so 
that  no  discounting  over  time  is  required. 
The  $140,400  cost  is  amortized  at  8-1/2%  for 
year  five.  Average  annual  costs  are  $35,662. 
Finally,  Table  6c  shows  that  the  proposed 
consolidation  plan  is  more  than  justified  from 
an  economic  perspective  with  a  benefit-to- 
cost  ratio  of  1 .60.  This  means  that  over  the 
five  year  horizon  the  action  will  return  $1.60 
for  each  $1.00  invested. 

Table  7c  shows  what  a  comparison  of 
alternatives  would  look  like.  Alternative  1  is 
the  plan  previously  discussed.  Other  plans 
show  varying  degrees  of  savings  and  costs 
but  allow  analysts  to  compare  different  ideas 


against  one-another  on  an  economic  basis.  It 
should  be  added  that  other  considerations 
aside  from  BCR's  may  need  to  be  considered 
as  well  but  this  approach  will  lead  to  the 
most  efficient  solution.  If  the  most  efficient 
solution  is  not  selected,  than  the  analyst 
knows  what  is  being  given  up  in  terms  of  lost 
benefits  to  obtain  another,  less  economic 
solution.  Note  that  the  net  gain  and  BCR  for 
alternative  1  are  the  highest  among  the 
alternatives  investigated. 

Figure  3c  shows  a  more  probable  or 
realistic  subprocess  of  an  organization.  It  is 
more  complex  and  may  have  loops  as 
illustrated  by  the  double  arrow  activities:  F- 
I,  I-H  and  H-F  where  processes  are  cycled 
back  and  forth.  Such  a  situation  may  occur 
in  COE  flood  control  formulation  planning 
when  hydrology  data  for  a  plan  are  given  to 
an  economist  who  then  develops  the  benefits 
of  the  plan,  which  then  goes  to  cost 
engineering  only  to  discover  that  costs  are 
too  high  and  the  projects  BCR  is  less  than 
unity.  The  process  is  repeated  until  a  plan  or 
set  of  alternatives  finally  evolves.  This  loop 
may  go  on  for  months  before  a  final  set  of 
alternative  flood  control  plans  are  sent  to  the 
next  event  or  activity. 

The  power  of  the  PERT  process  as 
described  in  the  previous  paragraphs  is  clear. 
Customer  driven  input  has  been  included, 
goals  have  been  established  and  met, 
objectives  have  been  fulfilled  and  a  process 
change  has  taken  place  and  proven 
justifiable.  Furthermore,  effectiveness  (doing 
the  right  thing )  has  been  integrated  through 
customer  input  and  efficiency  (doing  things 
right)  has  improved  because  costs  have 
decreased.  There  could  have  been  several 
alternatives  to  investigate  each  with  a 
different  benefit-to-cost  ratio  as  well  as 
meritorious  non-quantifiable  attributes. 
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TABLE  4c 

BENEFIT  (SAVINGS)  ANALYSIS  OF  PROPOSED  PROCESS  CHANGE 


EXISTING  PROCESS  COST: 
NEW  PROCESS  COST: 


(1) 

FUTURE 

PROJECTS 

NO. 


NET  SAVINGS  = 


(2) 

(3) 

YEAR 

EXPECTED 

NET  SAVINGS 

PRESENT  VALUE 
FACTOR  3 

1996 

$66,000 

0.921 

1997 

$60,000 

0.849 

1998 

$75,000 

0.783 

1999 

$35,000 

0.722 

2000 

$44,000 

0.921 

TOTAL  =  $224,981 
INTEREST  &  AMORT  =  0.254 
AVG.  ANNUAL  SAVINGS  =  $57,145 


$517,800 
451.800 
$  66,000 


PRESENT 

VALUE 


$60,786 


$50,940 


$58,725 


$25,270 


$29,260 


Interest  and  amortization  @8-1/2%  for  5  years 
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TABLE  5c 

COST  ANALYSIS  OF  PROPOSED  PROCESS  CHANGE 


COST  ITEM  1995  COSTS 


1.  Cost  of  meetings  &  strategy  sessions  $10,800 

2.  Cost  of  internal  administration  5,000 

3.  Cost  of  planning  8,000 

4.  Cost  of  research,  consulting  fees  10,000 

5.  Down-time  due  to  process  change-over  42,600 

6.  Cost  of  new  equipment  50,000 

7.  Re-train  employees  10,000 

8.  Cost  of  furniture  moving  2,000 

9.  Miscellaneous  expenses  2.QQ0 

TOTAL  $140,400 

Interest  &  Amortization 4  0.254 

Average  Annual  Costs  $35,662 


TABLE  6c  |j 

THE  BENEFITS  OF  THE  NEW  MANAGEMENT  ALTERNATIVE 

Average  Annual  Savings: 

$57,145  |j 

Average  Annual  Costs 

$35,662  jj 

Average  Annual  Net  Gain 

$21,483 

||  Benefit  to  Cost  Ratio: 

$57,145/$35,662  =  1.60  f  j 

4  Interest  and  Amortization  for  5  Years  @8-1/2%  Interest 
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TABLE  7c 

COMPARISON  OF  ALTERNATIVE  SOLUTIONS 


ALTERNATIVES 


COSTS/BENEFITS 

(1) 

(2) 

(3) 

(4) 

Avg  Annual  Savings 

$57,145 

$47,500 

$44,300 

$62,900 

Avg  Annual  Costs 

$35,662 

$37,000 

$39,900 

$49,700 

Avg  Annual  Net  Gain 

$21,483 

$10,500 

$4,400 

$13,200 

BCR 

1.60 

1.28 

1.11 

1.27 

non-quantifiable  attributes.  PERT  can  be 
used  to  explore  and  evaluate  projects, 
program  performance,  program  formulation, 
organizational  structure,  communication 
networks,  and  a  host  of  other  process  and 
systems.  It  is  also  useful  in  evaluating  the 
really  tough  issues  related  to  what  happens 
when  new  processes  cut  across  divisional 
boundaries.  One  of  the  reasons  process  and 
systems  analysis  is  necessary  lies  in  creating 
a  problem  when  finding  a  solution.  It 
sometimes  happens  that  in  removing  an 
obstacle  or  "bottleneck"  in  one  area  of  an 
organization,  another  bottleneck  is  created 
elsewhere.  Systems  analysis  helps  prevent 
this  from  happening  by  examining  all  the 
processes  involved;  not  just  the  problem  at 
hand. 


The  evaluation  can  be  integrated  with 
statistical  process  control  to  provide  valuable 
information  about  process  mean,  variance, 
reliability,  and  correlation  as  used  to  examine 
the  incremental  approach  to  process 
improvement  found  in  total  quality 
management;  or  restructuring  initiatives,  de¬ 
layering,  down  sizing,  and  the  more 
draconian  process  of  re-engineering.  This 
process  can  also  be  used  to  examine 
functional,  cross-functional,  and  vertical  and 
horizontal  flow  movement  in  the 
organizational  structure.  Best  of  all  there  are 
commercially  available  PERT  computer 
programs  which  can  be  used  with  the  office 
PC  to  perform  the  analyses  mentioned  above. 
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C  C  f  f  c 


3  3  3  3  3 


C.3  CONTROL  CHART 
TECHNIQUES 
_ [Rodakowski,  1994] 


Shewhart  Control  Charts 

An  excellent  discussion  of 
process  variation  and  statistical 
quality  control  can  be  found  in 
an  AT&T  publication  [AT&T,  1956]  from 
which  much  of  the  following  information  is 
based.  When  data  are  collected  relating  to  the 
measurement  properties  for  any  problem, 
production,  situation  or  process  (products  and 
services),  these  data  usually  display  random 
variation.  Instead  of  the  same  datum 
measurement  being  observed  each  time,  the 
measurement  (of  the  same  thing)  varies. 
When  these  data  are  plotted  one  can  observe 
an  up  and  down  or  nonuniform  saw  tooth 
character.  Similarly,  any  series  of  numbers 
from  a  process  will  produce  a  zig-zag  or 
fluctuating  pattern.  There  is  no  known 
product,  service,  item  or  process  which  does 
not  produce  or  show  measurement  variation 
over  time.  This  seemingly  trivial  observation 
is  extraordinarily  important. 

Measured  random  variations  are  caused 
by  a  large  number  of  reasons:  differences  in 
materials,  equipment,  the  local  environment, 
human  skills,  management  practices,  etc. 
Many  of  these  differences  are  small  but  are  the 
chief  driving  force  behind  fluctuations  which 
are  natural  or  normal.  On  occasion,  however, 
there  will  be  a  significant  or  unusual 
difference  more  profound  than  all  other 
differences  combined.  These  are  not  random 
process  variations.  Such  significant  dif¬ 
ferences  may  be  observed  when  equipment 
breaks  down,  employees  receive  new  training, 
management  changes  hands,  an  experienced 


worker  takes  over  where  an  inexperienced 
worker  left  off,  etc.  These  large  signatures  or 
causes  produce  a  fluctuation  pattern  which  is 
unusually  large  or  aberrant;  they  are  abnormal, 
but  they  are  not  random. 

Evidence  indicates  that  there  is  a  clear 
cut  and  measurable  difference  between  the 
natural  and  the  unnatural  fluctuations.  These 
differences  can  be  detected  and  subsequently 
analyzed  by  existing  statistical  tools  such  as 
the  frequency  distribution,  discussed  earlier. 
Once  it  is  determined  that  a  fluctuation  is 
unnatural,  its  cause  can  be  determined.  Thus, 
causes  can  be  isolated  and  studied  for  any 
process,  whether  it  be  a  production  process,  a 
service  output  process,  or  a  manufacturing 
process. 

Fluctuations  were  observed  long  ago  in 
the  manufacturing  of  carriage  wheels,  metal 
parts,  and  banking  services.  When  engineers 
measured  the  differences  between  the  same 
part  manufactured  in  the  same  way  within  the 
same  process,  they  found  that  the 
measurements  tended  to  cluster  around  a 
central  value  with  some  degree  of  scatter  on 
either  side  of  the  central  value.  This  is  the 
typical  pattern  of  a  frequency  distribution;  a 
subject  discussed  earlier.  If  the  cause  system 
is  constant,  the  frequency  distribution 
approaches  some  distribution  function;  a 
mathematically  predictable  behavior.  A 
consistent  or  repeated  pattern  is  formed  and  is 
made  up  of  a  large  number  of  fluctuations  - 
some  larger  or  smaller  than  others  -  shifting 
within  the  bounds  of  the  fluctuation  pattern 
itself  (bell  shaped  curve)  when  there  are  no 
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unusual,  significant,  or  abnormal  causes  at 
work. 

The  fluctuation  patterns  in  normal 
production  and  service  operations  have 
statistical  limits.  If  that  pattern  is  normal  or 
natural,  its  fluctuations  will  fit  within  the 
confines  of  these  limits  as  stated  earlier.  If  a 
pattern  is,  however,  unnatural,  its  fluctuations 
will  exceed  these  limits.  When  statistical 
limits  are  added  to  a  fluctuating  pattern  the 
results  are  referred  to  as  a  control  chart. 
These  statistical  limits  are  usually  3  standard 
deviations  from  the  mean.  Any  pattern  of  data 
in  a  control  chart  which  fluctuates  outside  of 
these  3  standard  deviations  from  the  mean  or 
which  show  non  random  (unnatural)  points,  it 
is  said  to  be  indicative  of  a  process  which  is 
“out-of-control”.  Three  standard  deviations  is 
a  generally  accepted  criteria  but  more  will  be 
said  about  this  later  in  the  text. 

The  control  chart,  in  sum,  is  a  group  of 
fluctuating,  random  patterns  representing  a 
process  bounded  within  statistical  limits.  The 
control  chart  is  the  back  bone  of  statistical 
process  control  and  capability  analysis 
mentioned  earlier.  There  are  two  types  of 
control  charts.  One  for  analyzing  variable 
characteristics  (anything  which  can  be 
measured)  and  one  for  attributes  (where 
something  is  produced  or  serviced  which 
either  passes  a  test  or  fails  it).  In  this  text  we 
will  be  dealing  chiefly  with  variable  control 
charts  because  they  fit  in  the  COE  process 
better  than  attributes  charts  and  provide 
considerably  more  information. 

The  notion  of  control  involved  taking 
action  with  the  intent  of  achieving  a  desired 
end;  a  means  -  ends  relationship.  The  prime 
purpose  in  statistical  process  control  is  to 
detect  and  eliminate  the  unnatural  fluctuations 
and  reduce  the  variation  in  natural,  normal 
fluctuations  to  foster  uniformity  of  quality.  In 


other  words,  product  or  service  quality  needs 
to  be  reproducible  within  limits.  Note  that 
when  a  significant  cause  is  discovered  and 
removed,  this  changes  the  control  limits  by 
narrowing  them.  Leaving  behind  more  typical 
cause  variation  fluctuating  in  a  narrower 
range.  The  process  of  detecting,  identifying 
and  eliminating  causes  may  be  a  long  run 
process.  Initial  rapid  elimination  may  occur, 
but  the  longer  term  or  more  complex  causes 
may  take  time.  The  idea  of  a  state  of 
statistical  control  serves  as  a  basis  for 
describing  and  reaching  the  goal  of  uniformity 
and  achieving  the  functional  capability  of  a 
process  capable  of  reaching  this  goal.  In  the 
following  paragraphs,  data  for  permit  reports 
will  be  used  for  a  step-by-step  control  chart 
construction.  More  will  be  said  later  with 
regard  to  the  concepts  of  special  and  common 
causes  of  variation. 

The  objective  of  statistical  process 
control  is  to  eliminate  special  causes  of 
variation  and  to  monitor  the  process  for  shifts 
in  process  averages  and  variances  over  time. 
This  goes  a  long  way  toward  improvement  in 
product  and  service  quality. 

As  discussed  earlier,  every  item,  service 
or  process  varies  in  some  characteristic. 
These  variations  may  be  due  to  human 
resource  capabilities,  environmental  factors, 
management  practices,  etc.  There  are  two 
major  types  of  variation  that  are  important  and 
distinct  to  process  capability  analysis.  They 
are:  common  cause  variation  and  special 
cause  variation. 

Common  cause  variation  is  the 
collective  effect  of  many  or  all  individual 
causes  of  variation  that  are  indigenous  to  the 
process  of  producing  a  good  or  service  that 
cannot  be  removed  without  management 
action.  Studies  have  shown  that  common 
cause  variation  is  85%  management  related 
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[Deming,  1982].  Each  of  the  many  causes  of 
variation  that  compose  the  total  may  exhibit 
vastly  different  distributions,  but  in 
combination,  they  are  approximately  normal. 
Consequently,  whenever  we  have  common 
cause  variation  we  can  assume  that  the  process 
population  distribution  of  the  variable  being 
measured  is  nearly  normal.  Table  8c  lists  a 
few  attributes  of  common  cause  variation  in 
the  COE.  This  list  should  be  studied  as  a 
means  of  building  a  sensitivity  to  the 
characteristics  of  common  cause  variation.  In 
order  to  correct  common  cause  variation,  that 
is  the  heart  of  a  nonuniform  production 
system,  corrective  action  is  required. 
Common  cause  variations  are  the  process 
errors  over  which  employees  have  no  control, 
i.e.,  for  the  COE  this  could  represent  changes 
in  regulations,  priority  changes,  equipment 
breakdown,  congressional  requests,  among 
others. 

Special  cause  variation  emanates  from 
individual  sources  of  variation  that  may  be 
statistically  identified  and  removed  from  the 
production  and  service  process.  When 
adequate  statistical  evidence  is  presented, 
employees  are  best  at  identifying  a  special 
cause  of  variation  and  are  paramount  in 
eliminating  the  cause.  Such  causes  include 
incomplete  or  improper  training  of  a  new 
employee,  a  piece  of  equipment  not  properly 
functioning,  an  incorrect  procedure  or 
technique,  etc.  These  problems  represent 
approximately  15%  of  process  variation 
[Deming,  1982],  Tables  9c  and  10c  list  a  few 
items  of  potential  common  or  special 
variation. 


The  keys  to  determining  whether  a  cause 
is  special  or  common  is  to  first  detect  it, 
isolate  it,  then  eliminate  it.  These  factors  fall 
under  the  topic  of  control  charts  which  are 
closely  allied  to  common  and  special  cause 
analysis. 

When  variable  data  are  evaluated,  the 
data  are  graphed  over  time  on  a  mean  chart 
and  a  range  chart.  For  each  sample  data,  the 
mean  and  standard  deviation  are  most  often 
used  for  plotting.  However,  the  range  may 
substitute  for  the  standard  deviation  because 
of  the  ease  of  its  determination  without  much 
loss  in  the  validity  of  the  final  results. 

The  initial  step  in  developing  a  control 
chart  is  the  assembling  of  a  sample  size. 
Sample  sizes  or  subgroups  may  be  as  low  as  2 
but  more  typically  are  5  or  any  number 
between.  A  sample  may  be  collected  weekly, 
monthly,  or  annually  depending  upon  the 
circumstances.  At  least  20  samples  are 
deemed  a  viable  size.  For  each  sample,  the 
mean  x  and  the  range  R  are  calculated,  then 
the  mean  of  the  individual  sample  means  are 
calculated  as  a  good  proxy  of  the  process 
mean.  The  expression  for  this  is: 

Equation  (1) 


x  =  2  Xj/fe 


where  k  _is  the  number  of  subgroups  or 
samples  (x  =  X-double  bar  or  the  average  of 
the  averages). 
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TABLE  8c 

COMMON  AND  SPECIAL  CAUSE  ATTRIBUTES5 

Special  Causes  Associated 

Common  Causes  Associated 

with  things  which  are: 

with  things  which  are: 

||  Unnatural 

Normal  || 

||  Disturbed 

Natural  | 

|!  Unstable 

Stable  || 

||  Non-homogeneous 

Unisturbed  .  | 

||  Mixed 

Homogeneous  || 

||  Erratic 

Coming  from  a  single  || 

||  Abnormal 

distributon  || 

||  Shifting 

Not  changing  || 

||  Unpredictable 

Steady  1 1 

||  Inconsistent 

Predictable  || 

||  Out-of-the-ord inary 

Same  || 

||  Different 

Consistent  || 

||  Important 

Statistically  constant  || 

||  Significant 

SBSB 

5  AT&T,  Statistical  Quality  Control  Handbook  (Indiana:  AT&T  Technologies,  Inc.  Sec.  Ed.  1958) 


1 
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TABLE  10c 

HOW  WOULD  YOU  RATE  EACH: 
SPECIAL  VS.  COMMON 


□  Major  unscheduled  tasks 


□  Flash  fires,  crisis  events 


□  Employment  freezes 


□  Priority  changes 


□  Hurricanes 


□  Loss  of  funding/budget  cuts 


□  Reorganization 


□  TQM 


□  Long  term  illness  of  key  employees/supervisors 


□  Changes  in  management,  supervision  or  leadership 


□  Major  policy  directions 


□  Congressional  requests 


□  People  leaving  (promotions,  other  jobs,  etc.) 
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Following  this,  the  average  range  of  the 
subgroups  are  calculated: 

Equation  (2) 


R  =  2  i?j/k 


where  k  is  the  number  of  subgroups  or 
samples  as  before. 

From  these  equations,  a  control  chart  can 
be  constructed  which  yields  a  significant 
amount  of  information  about  the  process 
under  investigation.  Given  earlier 
discussions,  we  can  expect  the  process  to  be 
normally  distributed,  or  nearly  so,  if  the 
driving  causes  of  variation  are  common.  The 
distribution  of  samples  sizes  themselves  are 
also  expected  to  be  normal  even  if  small 
samples  sizes  (2  through  5)  are  used.  One  of 
the  functions  of  using  the  mean  chart  is  to 
detect  a  change  in  the  production  or  services 
process  mean.  The  use  of  a  chart  for  this 
purpose  is  basically  a  hypothesis  test.  Meek, 
Taylor,  Dunning  and  Klafehn  [1987]  describe 
in  clear  terms  the  purpose  of  variable  control 
charts  which  is  summarized  in  the  following 
paragraphs.  If  a  sample  value  is  drawn  from 
a  process  operation  such  that  its  mean  is  larger 
or  smaller  than  the  overall  process  mean,  then 
it  is  likely  that  the  process  mean  has  changed 
position.  When  we  conclude  that  the  process 
mean  has  shifted,  when  actually  it  has  not,  we 
are  making  what  is  called  a  Type  I  error. 
Alternatively,  when  we  conclude  that  the 
process  mean  has  not  shifted,  when  in  fact  it 
has,  we  are  making  what  is  called  a  Type  II 
error.  Usually,  the  calculated  control  limits 


are  set  at  plus  or  minus  three  standard 
deviations  (3  s)  from  the  mean  to  limit  the 
chance  of  making  a  Type  I  error.  From  a  table 
of  standard  normal  distribution  Z-scores,  we 
can  ascertain  that  the  probability  that  the 
process  mean  has  shifted,  when  in  actuality  it 
has  not,  is  0.003  (3  standard  deviations). 
Figure  4c  illustrates  these  principles.  Three 
standard  deviations  are  used  because  studies 
have  shown  that  this  criteria  is  the  most 
economically  efficient  standard  to  use  when 
testing  Type  I  and  Type  II  errors  [Shewhart, 
1986]. 

It  is  the  conclusion  of  hypothesis  testing 
that  determines  the  three  standard  deviation 
control  limits  (3s).  Studies  conducted  in  the 
past  have  demonstrated  that  these  limits  are 
the  most  economically  efficient  criteria  when 
balancing  the  cost  and  benefits  of  detection 
and  problem  identification  against  the 
consequences  of  eliminating  a  special  cause. 

1.  Basic  Equations  for  Control  Chart 

Construction 

We  can  look  at  quality  control  from  the 
confidence  interval  for  X.  The  upper  and 
lower  control  limits  are  positioned  at  a 
distance  of  three  standard  deviations  from  the 
average  sample  mean  by  the  expression 
[Meek,  et  al,  1986]: 

Equation  (3) 


UCL-  =X+A2R 


Averages  (A  bar  chart) 
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TYPE  I  ERROR 


Equation  (4) 


LCT-=X  -  A,  R 


where  UCL,  and  LCL,  are  the  upper  and  lower 
control  chart  limits,  X  is  the  process  mean,  A2 
is  a  sample  size  adjustment  factor,  and  R  is  the 
mean  process  range,  and: 

Equation  (5) 


UCL-  =P4  R 


Ranges  (for  R  charts) 


Equation  (6) 


lcl7  =d3r 


where  UCLR  and  LCLR  are  the  upper  and 
lower  control  chart  limits,  R  is  the  mean 
process  range,  and  D3  and  D4  are  sample  size 
adjustment  factors.  Table  lie  shows  the  value 
needed  for  samples  size  adjustment 
multipliers. 

The  range  chart  R  is  used  in  conjunction 
with  an  X  chart  to  audit  the  variability  process. 
Variability  changes  are  just  as  undesirable  as 
changes  in  the  process  mean.  The  range  chart 
is  not  only  used  to  audit  process  fluctuation 
but  aids  in  identifying  ways  to  decrease 
process  variability.  One  of  the  functions  of 
statistical  process  control  is  to  reduce  the 


variation  around  the  desired  or  target  process 
mean. 

An  interesting  spinoff  use  for  control 
charts,  aside  from  monitoring  shifts  in  process 
mean  and  in  process  fluctuations,  is  the 
diagnosis  of  cause  and  effect.  Charting  aids  in 
detecting  a  problem  and,  along  with  other 
business  monitoring  tools,  can  go  a  long  way 
in  specifying  cause  and  effect  relationships. 
Simply  put,  more  is  learned  about  the  process. 

Figure  5c  illustrates  the  setup  needed  to 
analyze  the  behavior  of  the  process  mean  and 
examine  range  activity. 

2.  Rational  Subgroup  Selection 

A  subgroup  is  a  small  sample  size.  The 
choice  as  to  how  many  values  make  up  a 
subgroup  is  as  much  an  art  as  a  science. 
Generally  this  choice  is  dependent  upon  the 
behavior  of  the  phenomenon  being  chartered 
[AT&T,  1958].  If  the  raw  data  appears  to 
fluctuate  widely,  one  would  want  to  choose  a 
subgroup  size  around  4  or  5  to  help  moderate 
the  fluctuations.  If  the  fluctuations  appear 
dampened  already,  then  a  subgroup  size  of  2 
or  3  may  be  more  suitable.  You  do  not  want 
too  large  a  subgroup  size  as  this  would  tend  to 
disguise  the  magnitude  of  fluctuations  to  the 
point  of  hiding  special  causes.  Alternatively, 
a  sample  size  too  small  may  lead  to  many 
things  looking  like  special  causes.  Rational 
subgroups  are  usually  chosen  to  make  the 
variation  within  each  subgroup  as  small  as 
possible  for  the  process  (representing  the 
variation  from  common  causes)  and  so  that 
any  shifts  in  the  process  performance  (i.e., 
special  causes)  can  emerge  as  differences 
between  subgroups  [Shewhart,  1986].  There 
are  rules  of  thumb  for  selecting  subgroups. 
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They  include: 

□  A  subgroup  should  be  selected  in 
such  a  way  as  to  allow  for 
opportunities  in  variations  to  show 
up.  Consequently,  subgroup  size 
should  be  small  (2  to  5 
measurements  is  suitable). 

□  Subgroup  size  must  remain 
constant  for  all  subgroups. 

□  Data  should  be  taken  (sampled) 
frequently  enough  to  show 


variations  in  the  process  under 
investigation  -  give  the  process 
time  to  show  change 
fluctuations. 

□  The  number  of  subgroups  must  be 
sufficient  to  allow  process 
variation  to  show  up.  From  a 
statistical  point,  25  or  more 
subgroups  containing  about  100  or 
more  measurements  give  a  good 
test  for  stability  and  show 
variance. 


Wayne  W.  Daniel  and  Janies  C.  Terrell.,  Business  Statistics  For  Management  and  Economics.  5th  ed. 
Boston:  Houghton  and  Mifflin.  1989 
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Recall  what  was  discussed  earlier  in  the 
text;  that  the  samples  of  2-5  are  used  to  find  a 
mean  (X).  These  subgroup  means  are  then 
used  to  find  the  mean  of  the  means  (x).  The 
mean  of  the  sampling  distribution  of  means  is 
equal  to  the  population  mean.  We  also  find  the 
mean  of  the  ranges  R  (which  is  our  substitute 
for  the  standard  deviation)  approximates  the 
mean  of  the  population  range  distribution. 

An  example  of  how  to  develop  a  control 
chart  will  aid  considerably  in  understanding 
what  has  been  introduced  earlier. 

To  explain  the  process,  we  will  develop 
some  data  on  the  length  of  time  needed  to 
complete  39  Permit  Reports  during  an  FY  in 
a  Permit  Branch,  Construction  and  Operations 
Division.  The  report  number  and  length  of 
time  it  took  to  complete  each  permit  are  listed 
in  Table  12c.  As  the  data  show,  each  report 
took  different  lengths  of  time  to  complete 
ranging  from  a  low  of  60  days  to  a  high  of  120 
days  with  a  mean  of  95.5  days.  This  data  was 
then  broken  into  eight  class  intervals  in  order 
to  construct  a  frequency  distribution  of  the 
data.  This  process  is  illustrated  in  Table  13c. 
By  looking  at  the  tallies  in  the  table,  it  is 
readily  apparent  that  a  “bell  shaped”  curve  is 
formed  by  the  variation  within  the  data.  The 
number  of  reports  falling  into  each  class 
interval  are  presented  in  the  third  column  and 
support  the  tally  distribution.  This 
information  is  then  used  to  construct  a 
frequency  distribution  as  shown  in  Figure  6c. 
The  frequency  distribution  takes  the  expected 
form  of  the  standard  bell  shaped  curve  with  a 
range  of  values,  standard  deviation,  mean, 
mode,  and  median.  The  distribution  shows 
that  the  data  do  in  fact  vary  as  would  be 
expected.  This  is  the  type  of  variation 
discussed  earlier. 


Table  14c  shows  the  same  data  as  in 
Table  12c  but  grouped  into  subgroups 
(randomly  selected)  of  three.  Column  (1) 
shows  a  subgroup  number.  Each  subgroup 
number  consists  of  three  sequential  numbers. 
The  numbers  in  Column  (2)  represent  the 
values  used  to  form  that  particular  subgroup. 
In  this  case  the  subgroup  size  is  three 
numbers.  Remember  a  subgroup  can  consist 
of  between  2  and  5  numbers.  Remember  also 
a  subgroup  serves  two  purposes;  one  is  to 
dampen  oscillation  and  the  other  is  to  serve  as 
a  “sample”  of  a  larger  set  of  data.  Column  (3) 
represents  the  sum  of  the  values  in  each  group. 
The  values  in  parenthesis  in  Column  (4)  are 
rounded  numbers.  As  can  be  seen  at  the 
bottom  of  Column  (4),  x  (see  equation  1)  is 
calculated  as  95.9  days;  this  is  the  mean  of  the 
means.  Column  (5)  shows  the  range  R  of  each 
subgroup.  As  before,  the  range  is  the 
difference  between  the  highest  and  lowest 
value  in  a  group.  The  mean  of  the  ranges  (R) 
is  calculated  as  17.9. 

The  data  in  Table  14c  is  transferred  to 
Figure  7c.  Figure  7c  is  a  standard  form 
constructed  for  the  purpose  of  graphing  a 
control  chart  and  showing  all  the  information 
pertinent  to  the  development  of  the  chart.  It  is 
a  complete  record  in  and  of  itself.  Note  that 
the  data  is  located  in  the  bottom  left  of  the 
chart.  The  raw  data  are  shown,  the  mean  X  of 
each  subgroup  is  presented  and  the  range  R 
within  each  subgroup  is  also  shown. 

Figure  8c  shows  the  placement  of  process 
mean  report  completion  time  (x)  and  mean 
range  fluctuation  (R).  It  is  now  possible  to 
compare  movement,  variances,  or  fluctuations 
on  the  basis  of  a  frame  of  reference.  Figures 
9c  and  10c  add  to  Figure  8c  to  show  the 
graphing  process. 
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TABLE  13c 

DEVELOPMENT  OF  A  FREQUENCY  DISTRIBUTION 

Class  Intervals  of  the  Number  of  Days  Needed  to 
|  Complete  a  Class  Interval  of  Report  Permit  Application 

Tallies 

Number  of  Times  Each 
Completion  Was  Observed  || 
(Days)  (Source  Table  12c)  || 

51-60 

1 

■ 

2.6  || 

61-70 

1 

n 

2.6  || 

71-80 

11 

2 

5.1  || 

81-90 

mi  in 

8 

20.5  If 

91  - 100 

mi  nil  nil 

15 

38.5  || 

101-110 

mi  ill 

8 

20.5  || 

1  111-120 

in 

3 

7.7  || 

|  121-130 

l 

1 

2.6  || 

39 

100.17  || 

Mean  =  95.9  || 

Median  =95.0 

Mode  =  90.0 

Standard  Deviation  =  13.61 

i'ii'i'i'i't't'i'i’i'i'i'iii'iTiTi'i'i'i'l'i'i'i'i'i'i'i'i't'i’i'riiiiTi'i'i'i'i'i'tViViii'i't'i'i'ii'i'fiVi'*  mmnnmm . . . .  mini  niniiiitMMniiiiiiiiiiMUinitiiiiiiinnniiiiiiiiiiiiiiiimmiiiimninmiiiminmmnniiiinun  mn  in  nmnmnf 

7  Does  not  total  to  100%  due  to  rounding. 
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(1) 

Subgroup 

Number 


TABLE  14c 

ANALYSIS  OF  SUBGROUP  DATA 


(2) 

Values  Used  In 
Subgroup 
(No.  Of  Days) 


60+70+81 


130+81+77 


81+90+101 


90+103+110 


107+103+78 


100+90+90 


109+110+120 


108+90+111 


118+90+111 


93+94+95 


99+99+98 


97+92+100 


100+91+92 


(3) 

Sum  of 
Subgroup 
(Number  of 
Days)  x 


(4) 

Mean  of 
Subgroup 


70.7  (71) 


96.0  (96) 


90.7  (91) 


101.0(101) 


96.0  (96) 


93.3  (93) 


113.0(113) 


103.0(103) 


70.7  (71) 


96.0  (96) 


98.7  (99) 


96.3  (96) 


94.3  (94) 


13|  1247.2 


(5) 

Range  of 
Subgroup 

(R) 


R  =  17.9 
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Figure  6c. 
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Figure  7c.  -  Shewhart  Control  Chart 
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Figure  8c.  -  Shewhart  Control  Chart 


One  Team. 

One  Purpose... 
Dedicated  to  Excellence 
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IE 


T_M 


Figure  9c.  -  Shewhart  Control  Chart 
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Figure  10c.  -  Shewhart  Control  Chart 


Figure  11c  is  the  grand  finale.  Using 
Equations  3  through  6  we  have  calculated  the 
lower  and  upper  control  limits  for  the  process 
mean  x,  and  the  mean  process  range  R.  The 
equations  are  repeated  below  with  values 
inserted  so  that  the  reader  can  trace  how  the 
graphed  data  were  calculated.  The  equations 
are: 

The  location  of  the  Process  Mean  x: 

UCL  =  x  +  Aj  R  LCL  =  x-AjR 
=  95.9  +  (1.02X17.9)  =  95.9  -  (1.02)(17.9) 

=  114.2  =77.6 

The  location  of  the  Process  Range  R: 

UCL  =  DR  LCL  =  Dj  R 

=  (2.57X17.9)  =0(17.9) 

=  46  =0 

Note  the  sample  size  adjustment  factors 
are  on  the  figures  in  the  small  box  located  in 
the  upper  right  hand  side  of  the  graphs. 

Observe  that  the  points  located  outside 
the  3  standard  deviation  limits  are  denoted  by 
the  darkened  arrows.  These  points  represent 
an  out-of-  control  event  or  process;  they  are 
special  cause  induced.  The  special  causes  are 
two  independent  events  as  indicated  by  their 
different  subset  spawning.  Something  caused 
the  mean  value  to  fluctuate  so  widely  that  the 
process  mean  at  that  point  shifted  by  more 
than  3_standard  deviations  from  the  process 
mean  x.  Similarly,  something  in  the  range 
detection  indicated  that  a  special  force  caused 
the  range  to  widen  well  beyond  the  normal 
process  range;  i.e.,  past  3  standard  deviations. 
All  other  fluctuations  stayed  within  a  3 
standard  deviation  criteria. 

It  should  be  noted  that  the  shift  in  the  x 
for  the  special  cause  is  a  desirable  one;  the 
causer  should  be  found  because  this  point 


indicates  a  movement  in  the  direction  of 
decreasing  report  time  completion.  If  the 
point  had  moved  in  the  other  direction,  say  to 
130,  the  cause  would  need  to  be  determined 
because  this  has  the  tendancy  to  move 
completion  times  upward  —an  undesirable 
effect. 

Process  Analysis 

The  shift  noted  in  the  R  chart  is  undesir¬ 
able  because  whatever  caused  it  may  happen 
again  resulting  in  adverse  fluctuation  (a  wider 
spread)  in  the  range  of  the  production  process. 

1.  Deciphering  The  Magic  Behind  The  X 

and  R  Charts 

In  essence,  there  are  two  out-of-control, 
special  cause  related  conditions.  One  is  when 
X  and  R  data  fall  outside  their  respective 
statistical  limits  of  3  standard  deviations.  The 
second  occurs  when  a  pattern  emerges  with 
data  points  within  the  3  standard  deviation 
limits,  but  data  within  the  common  cause 
range  form  a  non-random  scatter 
configuration.  The  basic  Figure  12c  illustrates 
what  each  special  cause  might  look  like  if 
plotted  on  a  graph.  Note  that  these  patterns 
detect  a  problem;  it  takes  other  tools  (Pareto 
charts,  cause-and-effect  charts,  etc.)  to  isolate 
and  identify  the  problem.  Each  graph  applies 
to  both  the  X  and  R  charts. 

Figure  12c-l  has  been  the  major  topic  of 
concern  covered  in  previous  sections. 
Extreme  events  occurring  during  a  production 
process  (products  or  services)  have  resulted  in 
special  cause  events,  meaning  the  process  is 
out  of  control.  Two  out  of  17  points  have 
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Figure  11c.  -  Shewhart  Control  Chart 
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Figure  12c.  -  Out  of  Control  Patterns  for  X  and  R  Charts 
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Figure  12c.  -  Out  of  Control  Patterns  for  X  and  R  Charts  (continued) 
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exceeded  the  3  standard  deviation  limit;  one 
above  the  threshold  and  one  below.  This  is  an 
unstable  process.  The  causes  need  to  be  de¬ 
termined  and  eliminated  to  bring  the  process 
into  balance  or  into  stability.  The  cause  of 
such  an  event  could  be  one  or  multiple  and 
would  depend  upon  what  specific  process  is 
being  investigated.  There  is  a  strong  chance 
that  it  is  a  localized  or  work-environment 
related  problem  and  not  a  common  cause. 

Figure  12c-2  represents  an  out-of-control 
condition  which  forms  a  trend  line.  This  is  an 
out-of-control  process  because  the  data  points 
form  a  trend  which  is  a  pattern  that  is  not 
random.  By  definition,  a  pattern  is  not 
random.  Since  the  line  is  sloped  upward, 
something  in  the  process  is  getting  worse  over 
time.  This  could  be  due  to  a  piece  of  equip¬ 
ment  gradually  wearing  out,  resulting  in  the 
production  of  a  poorer  quality  product.  It 
could  also  represent  poorer  quality  reports 
being  produced  by  an  employee  whose  work 
load  has  substantially  increased  or  who  has 
been  given  assignments  which  frequently 
change  priority,  both  resulting  in  less  time  to 
do  quality  work.  Again,  the  cause  will  be 
dependent  upon  what  is  being  produced  and 
what  is  being  measured.  There  is  no  cookbook 
answer.  A  scatter  plot,  Pareto  chart,  flow  chart 
or  cause-and-effect  diagram  is  needed  to 
isolate  the  exact  cause  and  effect  relationship. 
Downward  sloping  trends  also  represent  a 
condition  of  special  causes.  Note  the  trends  in 
the  X  and  R  charts  in  Figure  11c.  We  have 
identified  another  out-of-control  problem. 

Figure  12c-3  shows  an  out-of-control 
process  called  a  cycle.  Again,  a  pattern  is 
formed  which  is  not  indicative  of  randomness. 
A  situation  like  this  could  be  caused  by  a 
budget  cycle  or  simultaneous  scheduling. 
Both  of  these  occur  when,  for  example,  a  host 
of  new  project  starts  are  all  funded  during  the 


same  time  frame  and  are  expected  to  follow 
the  same  processes  of  intermediate  reviews 
(IRC’s,  IPR’s,  etc.)  and  be  completed  at 
approximately  the  same  time.  This  causes  the 
“boom”  and  “bust”  cycles  illustrated  in  the 
graph.  It  could  also  be  caused  by  contracting 
or  inventory  cycles.  Despite  the  fact  that  all 
plotted  points  are  within  control  limits, 
observation  of  such  a  pattern  depict  non- 
random  situations  which  are  deleterious  to  the 
production  process  via  the  non-  uniformity 
and  low  quality  which  is  often  the  byproduct. 

Grouping  is  illustrated  in  Figure  12c-4. 
It  too  constitutes  an  out-of-control  production 
process.  Once  again,  the  random  criteria  is 
broken.  Grouping  can  be  caused  by  temporary 
over  control  by  an  individual  with  specific 
criteria.  It  may  also  emanate  from  over¬ 
control  by  influential  groups  who  have  a 
specific  target  in  mind  such  as  in  management 
by  objective  (MBO).  The  result  is  classified 
as  a  special  cause.  Micro  management  is 
often  a  culprit  as  well. 

Stratification  is  shown  in  Figure  12c-5. 
It  is  exemplified  by  product  and  service 
outputs  that  take  place  at  extremes;  as  shown, 
they  cling  to  the  upper  and  lower  control 
limits.  Stratification  can  occur  due  to 
rapidly  changing  priorities,  changes  in 
guidance,  and  as  part  of  a  moral  issue,  among 
others.  Whatever  the  cause,  there  is  a  pattern 
not  in  line  with  common  cause  variation. 

“Freaks”  are  nothing  more  than  highly 
unusual  events  which  have  an  extremely  rare 
event  occurrence.  Such  a  case  is  shown  in 
Figure  12c-6.  This  type  of  event  is  typified  by 
a  “bad  day”  or  a  “Peter  Principle”  happening. 
For  example,  ninety  percent  of  an  offices’ 
computers  are  stricken  by  a  virus  on  a  day 
when  a  congressman  needs  some  report, 
coupled  with  an  office  flu  taking  out  a  third  of 
the  staff,  garnished  with  five  new  flash  fires 
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and  a  discovery  that  some  vital  information 
needed  that  day  was  incorrect.  There  are 
plenty  of  reasons  for  a  setback  in  schedules. 
This  is  a  “Freak”.  It  is  random  or  highly 
unusual  and  may  never  happen  again  but  is 
nonetheless  an  out  of  control  situation  by  any 
definition. 

Sudden  shifts  in  the  process  are  depicted 
in  Figure  12c-7.  A  sudden  shift  is  also  not  in 
line  with  randomness  because  the  process 
mean  or  range  has  dramatically  shifted  from 
normal  to  abnormal  or  from  one  form  of 
abnormal  to  another.  These  shifts  can  be 
explained  by  changes  in  training,  shifting 
work  from  an  inexperienced  to  an  experienced 
worker  or  vice  versa.  Also,  changes  in  guid¬ 
ance,  management,  and  “flash  fires”  may  also 
result  in  sudden  shifts  in  process  performance. 

Figure  12c-8  shows  no  abnormalities. 
There  are  no  trends,  freaks,  stratifications,  etc. 
It  is  a  normal  process  governed  by  the 
randomness  of  common  cause  forces.  It  was 
placed  here  to  contrast  previous  special  cause 
patterns.  This  is  what  we  want  to  observe  in 
a  production  or  service  process.  All  causes 
are  common  causes.  It  shows  all  special 
causes  have  been  eliminated  leaving  only 
common  causes  to  tend  with. 

Hugging  is  a  pattern  depicted  when  the 
points  “hug”  the  centerline  or  one  of  the 
control  limits.  This  pattern  is  typical  of  micro 
management  techniques  where  fear  and  tight 
control  govern  the  process.  It  stems  from 
perceived  or  real  threats  if  production 
standards  exceed  the  upper  or  lower  control 
limits.  This  is  not  a  random  process. 

Runs  are  patterns  where  plot  points  stay 
in  one  position  or  another  for  periods  of  time 
as  in  Figure  12c-9.  Such  a  pattern  usually 
underscores  an  attempt  by  someone  to  follow 
a  service  or  production  process  which  is  not  in 
line  with  an  average  system  process.  Other 


reasons  exist  for  this  pattern,  but,  as  before,  it 
is  not  typical  of  a  common  cause  circum¬ 
stance. 

Finally,  the  reader  needs  to  be  cognizant 
of  multiple,  out-of-control  processes  as  shown 
in  Figure  12c- 10.  This  figure  shows  the 
simultaneous  problems  of  trends,  runs, 
hugging,  instability  and  sudden  shifts.  Many 
forces  are  at  work  producing  an  out-of-control 
process.  Some  of  these  forces  blend,  and 
subsequently  even  disguise  others.  They  must 
be  separated  and  eliminated  from  the  process. 

Figure  12c- 11  contains  a  variety  of 
anomalies;  a  trend,  stratification,  a  sudden 
shift,  and  signs  of  instability.  It  is  possible 
that  several  factors  may  show  up  on  a  control 
chart  resembling  this  figure. 

Table  15c  summarizes  the  various 
aberrations  which  are  indicative  of  adverse 
process  patterns. 

Process  Stability 

Previous  material  detailed  a  considerable 
amount  of  information  concerning  the 
structure  and  meaning  of  the  Xand  R  charts  in 
monitoring  product  and  service  output  quality 
and  uniformity.  There  are  a  few  simple  rules 
and  ideas  which  will  allow  for  a  relatively 
quick  detection  of  problems  as  they  appear  in 
control  charts.  They  include: 

□  A  process  is  out  of  control  if  7  points  in 
a  row  fall  below  or  above  the  center 
line  (Xor  R). 

□  A  process  is  out  of  control  if  7  points  in 
a  row  are  each  progressively  lower  or 
each  progressively  higher  than  the 
previous  points. 
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TABLE  15c  1 

TYPES  OF  OUT  OF  CONTROL  OR  1 

SPECIAL  CAUSE  PATTERNS®  f 

Instability 

Freaks  | 

Trends 

Sudden  Shifts 

Cycles 

Gradual  Shifts 

Grouping 

Hugging  ij 

Stratification 

Special  Runs  M 

□  A  process  is  out  of  control  for  any 
non-random  process  where  it  is 
easy  to  predict  where  the  next  point 
will  be. 

□  A  process  is  out  of  control  where 
any  non-random  process  is  present. 

□  A  process  is  out  of  control  where 
any  points  lie  outside  control  limits. 

Figure  13c  also  brings  to  light  a  few 
concepts  discussed  in  the  text.  Recall  that 
about  68  percent  of  all  data  points  must  fall 
within  one  standard  deviation  of  the  mean  for 
a  process  to  be  classified  as  normal.  This  is 
true  for  both  X  and  R  charts.  Roughly 
translated,  the  middle  third  of  a  control  chart 
frequency  distribution  should  contain  about 
2/3  (67%)  of  all  measurements  and 
approximately  1/6  (16%)  should  fall  into  each 
remaining  1/3  tail  of  the  curve.  This  rule  of 


thumb  helps  guarantee  normality  and  quickly 
identifies  skewness  in  a  process. 

Figure  14c  shows  the  pathway  to 
continuous  improvement  in  process  capability. 
An  explanation  of  the  techniques  starts  with 
Period  I  where  special  causes  are  first 
discovered  and  then  eliminated.  These  are 
identified  by  the  arrows  (instability 
conditionand  a  trend;  7  points  in  a  row)  and 
can  be  assessed  using  Figure  12c.  Period  II 
shows  that  all  special  causes  have  been 
identified  and  eliminated.  Note  that  as  we 
progressed  from  Period  I  to  Period  II,  the 
variance  of  the  system  has  decreased  and  the 
mean  process  declined  from  about  4.2  percent 
defects  to  about  2.2  percent  defects.  As 
common  causes  continued  to  be  eliminated 
through  action  to  the  system,  variance 
continues  to  decrease  and  the  mean  process 
defects  continue  to  fall  as  exemplified  by 
Period  II.  The  manager  or  employee  must 
continue  to  watch  for  new  special  causes  even 
if  the  old  system’s  special  causes  have  been 
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Figure  13c.  -  Symmetry  in  the  Output  Process 
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Figure  14c.  -  General  Pattern  of  Progress,  Dealings  “Constancy  of  Purpose1 


previously  eliminated.  The  reason  is  that 
things  change  over  time  and  new  special 
causes  not  previously  experienced  may  creep 
into  the  system.  When  special  cause  factors 
have  been  eliminated  (and  only  then),  we  are 
ready  to  contend  with  common  causes. 
Remember,  special  cause  is  responsible  for  15 
percent  of  system  process  problems.  The 
remaining  85  percent  are  common  causes 
which  are  chiefly  in  the  realm  of  management. 

Demos  Control  Charts 

Shewhart  control  charts  can  be  employed 
to  measure  the  performance  of  groups  of 
items,  as  shown  above,  and  entire  processes 
and  systems.  In  the  previous  example,  we 
examined  the  processes  of  writing  a  report. 
The  report  can  be  viewed  as  either  a  single 
task  or  a  process  composed  of  specific  groups 
of  employees  and  activities.  In  the  next 
example,  we  will  explore  the  use  of  Demos 
control  charts  in  following  the  progress  of 
multiple  tasks.  To  do  this,  we  will  use  modi¬ 
fied  Shewhart  control  chart  techniques.  The 
Demos  control  chart  is  another  tool  belonging 
to  the  family  of  statistical  process  control 
methods. 

Figure  15c  uses  the  Demos  control  chart 
to  analyze  and  trace  waste  (transit  time,  re¬ 
view  time,  waiting  time,  rework  time).  These 
were  evaluated  earlier  within  the  framework 
of  value-added  analysis  and  cycle  time.  The 
table  shows  a  team  of  30  workers  and 
managers  on  five  projects  or  studies  which 
will  ultimately  result  in  the  completion  of  five 
water  resources  reports.  A  district  office  task 
force  has  kept  records  on  a  monthly  basis 
recording  the  waste  and  process  time  for  these 
and  similar  reports  produced  during  previous 
assignments.  Such  records  will  prove  very 
valuable  as  we  will  soon  see. 


The  data  show,  as  expected,  that  real 
process  time  is  relatively  low  encompassing 
only  (1,420/7,200)  19.7  percent  of  the  30  day 
work  period.  The  data  indicate  that  the  five 
reports  fall  within  normal  statistical  para¬ 
meters  as  evidenced  by  the  historic  and 
current  average  hours  and  waste  encountered 
and  the  upper  and  lower  control  limit  of  250 
hours.  An  examination  of  the  table  for  causes 
or  low  process  time  leads  to  some  anomaly  for 
project  number  4  -  it  consumes  900  of  the 
1 ,340  hours  (67.2%)  alone.  The  Demos  chart 
located  below  the  data  shows  that  something 
went  wrong.  The  project  4  team  leader  is 
advised  of  the  problem  and  proceeds  to 
examine  his  control  chart  as  depicted  in 
Figure  16c.  His  current  average  for  internal  re¬ 
view  time  also  shows  a  value  of  180  hours 
which  is  above  the  upper  control  limit  of  170 
hours.  It  is  traced  to  hydrology  and  hydraulics 
work  with  an  internal  review  time  of  600 
hours!  The  advantage  of  the  Demos  control 
chart  approach  lies  in  being  able  to  trace  and 
record  impacts  along  the  hierarchy  to 
measureperformance,  to  trace  cause  and 
effect,  and  to  observe  multiple  events 
simultaneously.  This  can  go  a  long  way  in 
expediting  the  reduction  of  waste,  improve 
cycle-time,  and  increase  value-added. 

Control  charts  and  statistical  process 
control  fit  the  requirements  of  GPRA  and  the 
APIC  criteria  very  effectively.  They  are  a 
little  more  complex  to  understand  than  the  use 
of  ratio  analysis  but  provide  a  quantum  leap  in 
diagnostic  information  and  system 
performance  measurement.  Notice  that  this 
approach  measured  the  performance  of  a 
cross-functional  activity  and  its  team.  It  could 
have  been  used  to  measure  a  single  function, 
KRA  results,  program  performance  or  a  single 
project  as  well. 
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-WASTE  MEASURED  IN  HOURS- 
TEAM  =  30,  MONTHS 
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NOTE:  Basis  of  analysis  is:  30  FTE  x  30  days  x  8  hrs/dAY 

possible  hours. 

Figure  15c.  -  Demos  Control  Chart  -  Multi-Project  Use 
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■WASTE  MEASURED  IN  HOURS- 
PROJECT  4  TEAM 
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Figure  16c.  -  Demos  Control  Chart  -  Single-Project  Use 
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APPENDIX  D 


THE  MEASUREMENT  OF  EFFECTIVENESS 
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D.l.  THE  MEASUREMENT 
OF  EFFECTIVENESS 


The  Goals  and  Objectives  Approach 


Vmmlffectiveness  is  an  assessment  of 
AY  the  results  of  a  program 
&  J  compared  to  its  intended 
purpose.  Intended  purpose  is  most 
frequently  part  of  an  organization's  strategic 
plan  and  goals  and  is  expressed  in  the  form 
of  objectives.  Objectives  are  the  key  to 
measuring  effectiveness  performance.  Good 
objectives  have  six  characteristics  which 
include  [Management  Concepts,  Inc.,  1995]: 


□  They  state  exactly  what  result  is 
wanted, 

□  They  define  how  much  of  a  result 
is  wanted, 

□  They  indicate  the  finish  time  for 
the  desired  results, 

□  They  signal  where  the  outputs  are 
expected  to  occur, 

□  They  denote  who  the  results  are 
desired  for,  and 

□  They  describe  why  the  outputs  are 
needed. 

The  degree  to  which  these  objectives  can 
be  quantified  will  depend  in  part  how  they 
are  written.  There  are  circumstances  which 
do  not  lend  themselves  to  easy  measurement 
or  quantification  at  all.  It  is  best  to  set 
objectives  which  can  be  measured  to  prevent 
bias  in  interpreting  the  results  of  a 
performance  measurement  program.  In 
many  cases  judgement  also  plays  a  role. 
Measurement  also  tends  to  make  clear  what 


the  targets  are.  Let  us  set  up  an  example  of 
how  an  analyst  can  write  objectives  which 
are  measurable.  Table  Id  will  help  explain 
the  differences  between  well  written  and 
poorly  written  objectives. 

The  process  starts  out,  as  described 
earlier,  as  a  sequence  of  planning  events 
which  begin  with  a  vision  statement,  progress 
to  a  mission  statement,  continue  through  th e 
development  of  goals,  and  end  with  specific 
objectives.  All  of  these  statements  must  be 
aligned  with  the  same  theme  and  results  in 
mind.  Objectives  one  through  four  are 
designed  to  specify  the  details  of  GOAL  (3). 
Each  goal  would  require  its  own  set  of 
objectives,  but  for  simplicity,  and  because  it 
is  about  the  objective  at  hand,  we  will  focus 
chiefly  on  fulfilling  the  requirements  of 
GOAL  (3). 

The  first  objective  meets  all  the  criteria 
for  a  good  objective.  It  states  the  results 
desired;  it  develops  a  list  of  five  critical 
result  areas  and  ten  useful  index 
measurement  tools  and  techniques.  It 
specifies  how  much  is  wanted;  five  critical 
result  areas,  and  ten  tools  and  techniques.  It 
also  identifies  the  required  completion  time 
as  the  end  of  FY  1997.  The  results  are  to 
occur  for  each  division.  The  "who  for" 
question  points  to  district  use  and  GAO 
auditing,  and  the  reason  for  the  effort  is  to 
improve  cost  efficiency.  All  six  criteria  have 
been  satisfied.  Notice  that  in  fulfilling  all 
criteria  there  is  no  doubt  what  is  to  be 
accomplished,  for  whom,  when,  why  and  the 
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TABLE  Id 


(1)  The  purpose  of  this  objective  is  to  quantify  performance  from  an  efficiency 
perspective  for  use  by  the  district  and  GAO  auditors  to  evaluate  district  cost 
control.  Each  division  in  the  district  will  develop  a  list  of  the  five  most 
critical  cost  areas  (called  cost  drivers)  impacting  their  specific  operations 
using  Pareto  analysis,  and  institute  ten  index  measurement  tools  and 
techniques  that  quantify  process  operational  cost  efficiency  by  the  end  of  FY 
1997. 

(2)  With  the  same  purpose  specified  in  number  (1)  above,  each  division  will 
construct  a  business  process  map  down  to  the  branch  level  detailing  cycle  time 
and  dollars  expended  for  each  activity  node  in  the  process  map.  Use  a  five 
year  data  base,  PERT  techniques,  and  SPC  (statistical  process  control)  to 
arrive  at  a  mean,  median,  mode  ,  and  variance  for  each  activity  node.  This 
effort  will  be  completed  by  the  end  of  FY  1997. 


Objectives  to  Meet 

Goal  Number  (3): 
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PLANNING  Am  EFFECTIVE  OBJECTIVES 
continued 


Objectives  to  Meet  (3)  Using  the  purpose,  tools,  and  techniques  described  in  objective  number  (1) 

Goal  Number  (3):  above,  each  division  will  reduce  process  cost  per  employee-year  5  percent 

below  1997  inflation  adjusted  levels  by  the  end  of  FY  1997. 


(4)  Improve  the  quality  of  our  products  and  services  through  increased  cross¬ 
functional  coordination  thereby  reducing  operational  costs  in  the  long  run. 


targets  to  be  met.  This  is  not  a  "fuzzy" 
statement.  Can  the  results  be  measured? 

Suppose  that  the  five  critical  result  areas 
listed  are  completed  but  only  eight  of  the  ten 
index  tools  have  performed  as  hoped  by  the 
end  of  FY  1997.  When  the  situation  is  stated 
in  this  fashion,  it  is  possible  to  measure 
success  with  numbers.  The  case  sited  above 
would  indicate  that  performance  on  the  five 
critical  result  areas  was  100  percent,  but  that 
of  the  ten  tools  and  techniques  only  80 
percent  have  been  satisfied  (8/10=80%).  If 
the  1995-1997  deadline  was  missed  by  four 
months,  this  could  be  quantified  by: 
24+4/24=1.167  or  16.7  percent  over 
schedule.  A  performance  effectiveness  table 
for  the  objective  might  then  look  as  follows: 


Objective  Criteria 

Target 

Actual 

Performance 

Effectiveness 

Critical  cost  areas 

5 

5 

100% 

Performance  indexes 

10 

8 

80% 

Scheduled  Months 

24 

28 

83,3% 

The  above  process  demonstrates  that 
effectiveness  can  be  quantified  if  the 
objectives  are  written  in  the  appropriate  way. 
This  does  not  mean  that  the  process  is  not 


without  problems.  The  difference  between 
the  target  and  actual  objectives  do  not,  for 
example,  measure  the  relative  importance  of 
the  two  performance  indexes  not  developed. 
They  may  have  much  more  importance  than 
the  eight  that  were  developed.  This  type  of 
argument  is,  however,  true  with  any 
performance  measure.  There  are  no  perfect 
indicators;  absolutes  are  rare,  and  numbers 
are  not  absolutely  deterministic.  A  situation 
such  as  this  does  not  diminish  the  need  to 
quantify  nor  is  it  an  excuse  to  back  off  and 
resort  to  pure  judgement,  guesstimates,  or 
"shooting  from  the  hip"  management 
reactions.  Behavior  such  as  this  is  often  a 
favorite  tactic  for  those  who  are  concerned 
with  results  and  accountability. 

The  reader  can  examine  the  merits  of 
objectives  (2)  and  (3).  Test  them  against  the 
six  criteria  to  see  if  they  are  complete  or 
need  revision.  Also  test  them  for 
measurability.  These  objectives  were  written 
for  internal  district  performance  evaluation. 
Objectives  can  be  written  for  all  the  KRA's 
listed  earlier  in  this  document  and  for 
external  feedback  from  the  customer. 
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Objective  number  (4)  is  clearly  not  well 
written.  It  reeks  of  transitory  verbs, 
vagueness,  fuzziness,  and  breaks  just  about 
every  rule  of  a  good  objective.  A  recent 
review  by  OMB  of  early  GPRA  pilot 
performance  plans  by  numerous  agencies 
indicated  a  surprising  number  of  objectives 
written  similar  to  objective  (4).  Such 
objectives  constituted  about  80%  of  early 
"first  shot"  efforts  on  the  part  of  participating 
agencies  [Management  Concepts,  Inc., 1995]. 
Objectives  must  fulfill  the  criteria  listed  or 
will  be  returned  to  the  agency  unapproved 
and  for  further  work. 

The  Quality  Function  Deployment 
Approach  (QFD) 

The  Quality  Function  Deployment  (QFD) 
is  a  newly  developed  TQM  method  that 
allows  planners  and  managers  to  include 
customer's  wants,  needs,  and  expectations  in 
the  decision  process  [Berk,  1993].  It  is  a 
graphical  tool  for  making  decisions  where 
there  are  multiple  goals  and  objectives  and 
resulting  conflicts.  It  also  is  a  tremendous  aid 
in  blending  customer  expectations  with  the 
engineering  and  planning  process  of  the  COE 
to  form  the  desired  outputs  and  outcomes.  In 
this  respect  it  is  an  effectiveness  process  tool 
and  an  effectiveness  measurement  tool.  It 
employs  the  "voice  of  the  customer"  in  what 
we  do.  It  began  as  an  engineering  tool  used 
to  match  customer  needs  and  wants  with  the 
design  and  engineering  process  in  the  private 
sector  but  it  became  valuable  as  a 
"marketing"  tool  for  public  organizations. 
This  is  novel  because  in  times  past  public 
institutions  have  not  had  a  market  tool  other 
than  the  public  meeting.  It  incorporates 
cross-functional  team  efforts  to  help  eliminate 


"surprises"  when  products  and  services  cross 
from  the  planning-engineering  phase  of  work 
to  the  public  review  phase  of  planning  where 
customers  examine  the  plans  for  themselves. 
This  is  often  a  difficult  and  expensive  event 
as  customers  find  out  that  what  they  wanted 
is  not  what  is  on  paper.  Considerable  re¬ 
formulation  becomes  necessary.  This 
situation  is  depicted  in  Figure  Id.  With  QFD 
this  is  avoided,  product  and  service  quality 
improves,  waste  (as  described  earlier)  is 
reduced,  and  tax  dollars  are  saved. 

For  this  to  be  an  effective  tool,  the 
decision-makers  must  be  cognizant  of  the 
importance  of  systems-related  operations. 
That  is,  most  production  and  service 
problems  are  not  caused  by  employee 
activities  but  by  systems  problems.  The 
organization  must  be  willing  to  adopt 
statistical  process  control  as  described  earlier 
and  design  of  statistical  experiments.  These 
are  the  tools  which  form  the  foundation 
which  connects  the  COE  management 
process  and  the  customer  making  the 
organization  much  more  effective.  To  be 
successful  today  we  must  be  at  least  as 
customer  driven  as  we  are  engineering 
driven.  Also  notice  that  QFD  is  a  prevention 
tool ;  not  a  post  problem  analysis  tool.  It  fits 
quite  well  with  the  Deming  philosophy  of 
building  quality  in  the  process  is  rather  than 
inspecting  it  out-after  the  fact.  It  also  a 
navigation  aid  in  making  decisions  that 
require  trade-offs  between  conflicting  needs, 
wants  and  customer  expectations.  One  of  the 
more  powerful  aspects  of  QFD  is  its  use  with 
benchmarking  to  compare  internal 
organization  perceptions  with  those  of  the 
customer  and  competitors. 

One  of  the  questions  QFD  addresses  is 
that  of  determining  the  WHATs  of  customer 
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Figure  Id.  -  Japanese  and  U.S.  Engineering  Design  Changes 


needs  and  expectations.  An  example  of  this 
presentation  is  illustrated  on  the  right  side  of 
Figure  2d.  In  this  example  five  WHATs 
were  requested  by  the  customer.  This 
information  was  obtained  through  customer 
surveys  and  interviews.  For  a  flood  control 
project  the  customer  needs  requires  five 
WHATs  including:  high  project  economic 
benefits,  environmental  sensitivity,  a  100 
year  level  of  protection,  the  lowest 
reasonable  cost,  and  it  must  be  acceptable  to 
their  general  constituency.  The  interviews 
used  to  assemble  this  information  are  used  to 
elaborate  and  clarify  WHATs  but,  just  as 
importantly,  to  unmask  any  unspoken  or 
unexpressed  desires  not  listed.  It  is  better  to 
delight  customers  than  to  just  satisfy  them. 
This  is  a  quality  touch  that  incorporates 
unexpressed  wants  in  the  WHAT  matrix.  In 
simple  terms,  the  WHAT  matrix  is  a  conduit 
for  the  wants  of  the  customer. 

The  next  step,  after  determining  the 
WHATs,  is  to  ascertain  the  HOW  much  of 
the  WHATs  can  be  achieved.  "The  HOWs 
are  what  an  organization  has  to  implement  in 
order  to  satisfy  the  WHATs"  [Berk,1993]. 
These  are  also  shown  in  Figure  2d.  Some  of 
the  HOWs  may  conflict  with  one-another; 
others  may  complement  each  other.  The 
HOWs  often  fall  under  the  engineers 
umbrella  because  they  are  design  and 
performance  oriented.  In  this  case  the  most 
desirable  HOWs  from  an  engineering 
perspective  must  be  mixed  with  those  from 
the  customer's  perspective.  The  engineer 
must  be  willing  to  accept  customer  features 
because  the  customer  is  the  one  paying  the 
bills  and  using  the  final  output.  This  does 
not  mean  that  safety  is  compromised  but  that 
the  "best"  engineering  designs  are  not 
necessarily  preferable  to  customer  wants.  It 


is  clear  that  any  HOW  can  help  satisfy  more 
than  one  WHAT  and  sometimes  they  are  in 
conflict;  this  is  an  expression  of  what 
happens  in  the  real  world.  Also  HOWs  can 
conflict  or  be  contrary  to  each  other  as  well. 
This  is  a  desirable  situation  because  it  is 
important  to  observe,  document,  and  weed 
out  conflicts  within  the  HOWs  now  rather 
than  find  problems  after  time  and  funds  have 
been  expended.  As  the  table  shows  it  is 
convenient  to  list  the  WHATs  in  a  column 
and  the  HOWs  in  a  row  to  form  a  matrix 
which  has  blank  areas  or  squares  which  can 
be  filled  in  with  more  information  about  the 
relationship  between  the  WHATs  and 
HOWs.  As  alluded  to  above,  the  blank 
spaces  in  the  WHAT-HOW  matrix  can  be 
used  to  draw  a  strength  or  correlation 
relationship  between  the  WHATs  and 
HOWs.  This  matrix  shows  the  manner  in 
which  the  HOWs  affect  the  WHATs.  The 
legend  shown  in  the  upper  left  hand  comer  of 
the  matrix  shows  the  traditional  symbols 
used  to  describe  the  degree  of  relationship 
between  the  WHATs  and  HOWs.  A  circle 
with  a  dot  in  the  center  indicates  a  strong 
relationship.  A  simple  circle  shows  a 
moderate  relationship.  A  triangle  indicates  a 
weak  relationship.  These  symbols  show  how 
strongly  a  HOW  supports  the  achievement  of 
a  WHAT.  A  blank  square  indicates  no 
relationship.  Such  an  entry  suggests  that  a 
HOW  has  been  incorporated  that  does  not 
satisfy  any  WHATs  which 
means  it  can  be  deleted  to  reduce  costs;  a  first 
important  step  in  reducing  process  costs.  It 
may  also  mean  that  a  customer  need  (a 
WHAT)  is  not  being  met  in  which  case 
additional  analysis  is  needed.  The  purpose 
thus  far  in  constructing  the  QFD  matrix  is  to 
match  customer  WHATs  with  design  and 
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Figure  2d.  -  Measures  of  Effectiveness  -  Quality  Function  Deployment  (QFD)  Matrix 
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planning  concepts— to  produce  a  product 
which  meets  or  exceeds  an  array  of  customer 
needs.  This  is  an  effectiveness  criteria. 

It  may  be  useful  to  elaborate  on  the 
symbols  within  the  matrix.  Let  us  take  the 
WHAT  of  Environmental  Sensitivity  to  show 
how  to  use  this  matrix.  The  HOW  of  Earth 
Channels  shows  a  strong  relationship  with 
environmental  sensitivity.  This  is 
understandable  especially  because  an  earth 
channel  compared  to  a  concrete  channel 
solution  to  a  flood  control  problem  is 
generally  regarded  by  environmentalist  as 
preferable.  Similarly,  a  dry  detention  is 
strongly  related  to  an  environmental  WHAT 
because  it  creates  recreation  space  (can  be 
used  as  a  park  or  recreation)  and  it  would 
prevent  urban  development  which  may 
otherwise  increase  run-off  and  flooding 
downstream.  It  also  maintains  the  integrity 
of  open,  green  space  and  habitat.  In  this  case 
the  HOW  of  levees  is  only  weekly  tied  to 
environmental  sensitivity  because  any 
channel  work  at  all  (earth  or  concrete)  will 
require  that  the  ambient  channel  and  right-of- 
way  habitat  conditions  be  disturbed.  On  the 
positive  side,  a  levee  may  create  a  buffer 
between  disturbing  influences  from  activities 
on  the  urban  side  of  the  levee.  In  this  case 
the  HOWs  of  concrete  channels  and  property 
buy-outs  have  little  or  no  correlation  with 
environmental  sensitivity  and  are  left  blank. 
The  responses  to  each  of  the  rows  and 
columns  in  the  QFD  matrix  will  likely  vary 
for  each  project  and  what  was  a  satisfactory 
answer  in  one  case  may  not  be  in  another. 

There  is  a  HOW  MUCH  associated  with 
the  HOW  which  helps  determine  success  of 
the  design  and  planning  teams  in  meeting 
customer  expectations.  This  is  illustrated  at 
the  bottom  of  the  matrix  in  Figure  3d  .  This 


portion  of  the  matrix  provides  a  glimpse  of 
what  customers  consider  as  satisfactory  in 
meeting  their  needs  and  wants.  For  example, 
under  the  column  labeled  earth  channels  in 
HOW,  customers  would  be  satisfied  if  80 
percent  of  all  required  channel  work  were 
earth  noting  that  in  some  circumstance  some 
concrete  use  may  be  necessary  to  preserve 
channel  integrity,  around  bridge  abutments, 
along  bends  in  the  channel,  etc.  The 
remainder  of  the  HOW  MUCH  section 
describes  similar  requirements.  These 
requirements  are  also  determined  by 
customer  surveys  and  interviews. 

Figure  4d  shows  yet  another 
characteristic  of  the  QFD  matrix.  The 
triangle  located  on  top  of  the  matrix  is  what 
is  called  a  CORRELATION  MATRIX  and 
has  its  own  legend  located  in  the  upper  tight 
hand  corner  of  the  figure.  This  part  of  the 
matrix  is  important  in  detecting  conflicts 
between  the  HOWs.  A  circle  with  a  dot  in 
the  center  means  a  strong  positive 
correlation.  A  simple  circle  means  a 
moderately  positive  conflict.  A  #  and  ## 
represent  negative  and  very  negative 
correlations  respectively  between  the  HOWs. 
This  matrix  allows  us  to  discover  not  only 
what  the  conflicts  are  but,  to  determine  the 
degree  of  conflicts  that  may  exist  in  the 
process.  Resolving  these  conflicts  will  often 
require  trade-offs  and  prioritizing  of  wants 
and  needs— compromise.  Many  of  the 
conflicts  do  not  necessarily  require  that  a 
HOW  be  dropped  in  the  trade-off  process  but 
may  be  dealt  with  by  modifying  the  HOW 
MUCH  portion  of  the  matrix.  The  trade-offs 
are  customer  driven  but  are  done  in  con¬ 
sultation  with  the  engineers  and  planners 
from  the  COE.  Notice  that  the  use  of  earth 
channels  and  concrete  channels  are  in  conflict 
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(strong  negative  correlation)  in  the 
CORRELATION  MATRIX  within  the  HOW 
and  with  the  WHAT.  The  reason  for  this  was 
discussed  earlier;  the  locals  want  earth 
channel  solutions  and  not  concrete  channels 
due  to  the  environmental  aspects  of  earth 
channels.  This  is  a  point  of  trade-off  that 
needs  to  take  place.  The  solution  may  lie  in 
HOW  MUCH  in  terms  of  the  percent  of  earth 
channels  and  the  percent  of  concrete  channels 
used. 

Figure  5d  completes  the  matrix  by  the 
inclusion  of  two  benchmarking  components; 
internal  benchmarking  and  external 
benchmarking.  The  purpose  of  these  bench¬ 
marking  analyses  are  to  compare  customer 
perceptions  of  what  we  perceive  as  good  and 
bad  project  attributes  and  what  the  customer 
perceives  as  good  and  bad  attributes.  This 
serves  as  a  reality  check  on  plans  and  designs 
and  helps  assure  that  the  systems  is  customer 
driven  rather  than  engineer  driven.  These 
ratings  can  be  obtained  with  customer 
surveys  and  through  interviews. 

The  external  benchmarking  process  uses 
a  CUSTOMER  BENCHMARKING  process 
and  is  located  to  the  right  of  the  matrix  in 
Figure  5d.  Notice  the  area  which  is  labeled 
"BAD"  and  "GOOD".  This  is  filled  in  by 
the  customer  after  consultation  with  the  COE 
on  engineering  and  planning  characteristics. 
The  "U"  represents  the  COE  effort,  the  "A" 
and  "B"  are  competitor  designs  as  viewed  by 
the  customer.  The  lines  connecting  the  "U" 
rankings  represent  the  graph  of  COE  per¬ 
formance.  Notice  that  the  customer's 
perception  of  our  plans  are  good  for  the 
WHATs  in  project  benefits,  and  level  of  pro¬ 
tection  but  are  bad  on  the  cost  of  construction 
side.  In  essence  this  shows  how  well  the 
customers  feel  the  designs  and  plans  satisfy 


their  needs;  it  represents  feedback  to  the 
COE. 

Internal  benchmarking  can  be  observed 
at  the  bottom  of  the  matrix  just  below  the 
HOW  MUCH  portion  of  the  matrix.  It  is 
entitled  PLANNING  AND  ENGINEERING 
BENCHMARKING.  The  symbols  are  the 
same  as  with  external  benchmarking.  This 
benchmarking  effort  is  done  internally  and 
includes  an  analysis  of  competitor  ideas  and 
plans  compared  to  the  ones  developed  by  the 
COE.  Notice  that  the  COE's  impression  of 
its  plans  and  designs  are  perceived  to  be 
better  than  the  customer's  perceptions  of  our 
plans  and  designs.  This  acid  test  is  needed 
to  satisfy  the  domain  of  customer  wants  and 
needs. 

The  QFD  matrix  has  one  more  valuable 
characteristic  to  contribute.  This  is  the 
IMPORTANCE  RATING  located  at  the  very 
bottom  of  the  table.  It  is  developed  in  the 
following  manner.  The  customer  is  asked  to 
rank  each  WHAT  based  upon  how  important 
that  WHAT  is  to  them.  This  is  now  shown 
next  to  the  WHAT  characteristics;  to  the 
right  of  the  matrix  with  the  numbers— 
4, 4, 3, 5, 2. 

Each  rating  is  assigned  an  importance 
number  as  shown  in  the  legend  below  the 
WHAT  portion  of  the  matrix.  Each  row  and 
column  entry  in  the  WHAT-HOW  matrix  is 
multiplied  by  the  importance  scale  number  to 
arrive  at  an  IMPORTANCE  RATING 
located  at  the  very  bottom  of  the  figure.  For 
example: 

□  the  4  next  to  the  High  Project 
Benefits  under  WHAT  is 
multiplied  by  the  value  in  the 
triangle  of  1  (in  the  legend)  to 
arrive  at  a  value  of  4; 


Appendix  D  -  The  Measurement  of  Effectiveness  - 165 


Correlation  Legend 
O  Strong  Positive 


Figure  5d.  -  Measures  of  Effectiveness  -  Quality  Function  Deployment  (QFD)  Matrix 


Appendix  D  -  The  Measurement  of  Effectiveness- 166 


□  the  4  next  to  Environmental 
Sensitivity  under  WHAT  is 
multiplied  by  a  value  of  10  to  arrive 
at  a  value  of  40; 

□  the  3  next  to  100  Year  Protection 
under  WHAT  is  multiplied  by  a 
value  of  1  to  arrive  at  3; 

□  the  5  next  to  Low  Cost  under 
WHAT  is  multiplied  by  10  to  arrive 
at  50,  and; 

□  the  2  next  to  Socially  Acceptable 
under  WHAT  is  multiplied  by  10  to 
get  20. 

The  values  are  summed  by  column  to 
arrive  at  an  Importance  Rating  of  117  (4  + 
40  +  3  +  50  +  20  =  117).  The  same 
process  is  done  for  each  of  the  five  columns. 
The  resultant  values  are  used  to  measure  the 
importance  of  items  in  the  relationship 
between  WHAT  and  HOW.  The  numbers 


have  no  particular  meaning  in-and-of 
themselves  but  convey  relative  importance 
for  quality  assessment  and  in  making  trade¬ 
offs,  if  needed. 

The  QFD  matrix  helps  identify  customer 
needs  and  expectations,  determines  how  to 
meet  them,  establishes  goals  and  objectives, 
leads  to  improved  quality  and  quantifies 
effectiveness.  This  is  precisely  the  tool 
needed  to  satisfy  NPRA  and  the  APIC 
criteria.  It  focuses  work  efforts  on  the 
customer,  provides  good  project  analysis 
opportunity,  improves  program  performance, 
guides  organizational  management,  and 
prevents  team  members  from  going  astray  of 
what  is  important.  It  allows  for  the 
development  of  plans  with  customer  input 
well  before  large  sums  of  money  are 
expended  and  prevents  much  of  the  waste  of 
re-work  (reformulation),  and  reduces  the 
cycle  time  (doing  it  right  the  first  time)  often 
accompanying  the  traditional  COE  approach 
to  planning  and  engineering. 
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E.l.  SURVEYS  & 
PARETO  CHARTS 


Surveys 

This  part  of  the  text  will  not  cover 
the  subject  of  customer  surveys 
in  any  comprehensive  format. 
The  subject  is  so  vast  that  it  would  require  an 
entire  book  to  do  it  .  Rather,  the  following 
material  will  focus  on  surveys  which  are 
quantifiable  in  character  rather  than 
narrative.  It  will  further  zero-in  on  those 
types  of  survey  questions  which  will  yield  a 
Pareto  chart  or  a  probability  frequency 
distribution.  These  are  very  powerful  tools 
and  satisfy  the  subject  of  this  text- 
performance  measurement.  It  is  recognized 
that  narrative  responses  to  survey  questions 
are  very  important  and  should  be  a  part  of 
any  comprehensive  survey  taken  to  determine 
values,  aesthetics  and  customer’s  feelings  and 
attitudes  about  what  impacts  them.  The 
subjects  of  internal  and  external  customers 
will  be  covered,  although  the  quantification 
of  the  survey  of  either  of  the  two  is  virtually 
the  same.  The  subject  is  highly  topical  in 
terms  of  GPRA  and  APIC.  Both  documents 
strongly  suggest  that  surveys  are  an  integral 
part  of  performance  measurement  and 
analysis  and  will  go  a  long  way  in  helping  to 
win  the  new  Baldrige  (APIC)  criteria  in  the 
ACOE  award. 

An  internal  customer  is  anyone  in  your 
organization  for  whom  you  provide  a  service 
or  product.  To  illustrate,  assume  you  are 
part  of  the  transformation  process  and  add 
value  to  the  services  you  provide  or  the 
products  you  make.  Someone  else  receives 


this  output  service  or  product  from  you 
within  the  organization  and  adds  further 
value  in  what  they  do,  and  so  on.  If  what 
you  pass  on  to  the  next  person  in  the 
production  process  is  of  poor  quality,  they 
must  contend  with  this  situation,  either  by 
returning  it  to  you  or  by  correcting  it 
themselves.  The  alternative  is  simply  to  pass 
the  error  on  to  the  customer.  By  the  time 
this  error  is  caught  somewhere  along  the 
transformation  chain,  we  either  have  to 
rework  the  problem  (time  is  wasted,  costs  are 
incurred),  or  an  unhappy  customer  returns 
the  flawed  item.  Any  scenario  under  these 
circumstances  results  in  a  costly  situation. 
This  means  that  any  person  who  receives  a 
faulty  item  or  service  you  provided, 
represents  an  unhappy  customer;  this  is  your 
internal  customer. 

An  external  customer  is  the  recipient  of 
the  final  product  or  service.  It  is  the  person 
or  persons  who  pay  for  the  output  or  item 
with  certain  performance  expectations, 
particular  wants  and  needs.  If  errors  are 
present  or  the  product  or  service  flawed  in 
some  way,  an  unhappy  customer  is  the 
outcome.  The  consequence  may  be  the  loss 
of  a  customer,  returned  merchandise, 
rework,  wasted  time,  increased  costs,  etc. 

There  are  tools  which  can  be  used  to 
measure  either  internal  or  external  customer 
satisfaction.  Surveys  are  one  vital  tool  for 
this  purpose  and  Pareto  charts  are  another. 
They  are  both  critical  forms  of  two-way 
communication  with  feedback.  One  tactic  in 
utilizing  surveys  is  to  have  the  customer 
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construct  the  questions  for  you— of  the 
questions  they  want  to  be  asked  about  your 
performance.  Another  is  to  have  an 
uninterested  third  party  such  as  a  consultant 
or  university  produce  a  scientific,  unbiased 
questionnaire. 

Surveys  take  various  forms  including 
written  questionnaires  from  one-on-one 
interviews,  written  questionnaires  from 
mailed  surveys  and  telephone  questionnaires. 
There  are  closed-ended  questions  and  open- 
ended  questions.  Whatever  method  is  chosen 
the  one  growing  preference  lies  in 
quantifying  results  so  that  they  can  be  used  to 
draw  conclusions  and  improve  organizational 
performance.  The  days  of  making  decisions 
from  “intuition”  and  “judgement”  are  rapidly 
disappearing.  There  is  still  a  surprising 
amount  of  this  going  on  today.  The  most 
significant  aspects  of  using  surveys  for 
obtaining  information  are  threefold;  to  obtain 
data  or  feedback,  to  use  this  feedback  to 
make  system  improvements,  and  to  construct 
the  survey  in  an  unbiased  manner.  Though 
you  may  wish  to  construct  the  survey 
questions  yourself,  it  is  advisable  to  have  a 
contractor  review  them  to  prevent  leading 
questions,  directional  wording,  and  add 
clarity  if  needed.  This  is  an  insurance  policy 
to  prevent  bias  from  entering  the  survey  and 
tainting  the  results.  It  is  also  a  factor  APIC- 
reviewers  will  look  for. 

Surveys  can  serve  a  variety  of  functions: 
to  gather  opinions,  probe  attitudes,  gauge 
climate  and  identify  opportunities.  They  are 
also  a  cost-effective  way  to  gather 
information  and  measure  progress.  The  best 
way  to  illustrate  the  use  of  surveys  is  to 
provide  an  example  of  the  use  of  a 
questionnaire.  Table  le  contains  a  list  of 
quoted  comments  [Heaphy  &  Gruska,  1995] 


of  critical  potential  customer  responses.  The 
questionnaire  or  survey  is  directed  toward 
internal  customers  for  management 
purposes.  It  is  a  questionnaire  designed  to 
obtain  “employee  perceptions”  about  a 
TQM/APIC  effort  going  on  within  their 
organization.  The  questions  themselves  can 
be  re-worded  for  different  effects.  They  can 
also  be  modified  for  external  customer 
responses.  Notice  that  some  questions  allow 
five  responses  and  others  six.  It  is  a 
generally  believed  notion  that  too  few  choices 
(say  3  or  4)  to  question  responses  may  not 
cover  the  range  of  potential  answers  leaving 
the  respondent  a  little  frustrated.  In  like 
manner,  too  many  choices  in  many  cases 
require  a  level  of  discrimination  not  easily 
dealt  with  by  respondents.  For  example,  a 
ten  gradation  response  requires  a  very  fine- 
tuned  thought  process  on  the  part  of  the 
respondent.  This  may  be  frustrating  as  well. 
It  is  a  good  idea  to  allow  a  response  range  of 
between  5  and  7  choices  although,  in  some 
cases,  this  rule  of  thumb  can  be  broken. 

Let  us  take  question  one  from  Table  le 
regarding  “customer  driven  quality”  as  an 
example  of  application.  Table  2e  and  the 
accompanying  figure  (Figure  2e)  located 
below  represents  an  analysis  of  responses  to 
the  hypothetical  survey.  Column  one 
represents  the  coded  response  circled  by  the 
respondent.  Column  two  is  the  response. 
Column  four  is  the  percent  breakout  and 
column  five  shows  how  the  mean  response  is 
calculated.  Forty  nine  respondents  addressed 
the  questions  with  a  resulting  mean  of  2.89 
(2.9,  rounded).  The  largest  interval  of 
responses  resided  with  number  three, 
“moderate”,  which  is  supported  by  the  mean 
response.  The  figure  below  depicts  the 
frequency  distribution  of  the  data  above. 
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TABLE  le 

CULTURAL  CHANGE  SURVEY  PERCEPTIONS2 


“  Customer-driven  quality :  The  ultimate  goal  of  quality  efforts  is  to  delight  the 
customer.  This  is  a  strategic  concept  that  requires  a  company  to  pleasantly  surprise  and 
delight  the  customer,  not  just  prevent  defects.”  Are  we  accomplishing  this  goal? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


2.  “Leadership  involvement:  The  senior  executives  of  a  company  establish  the  company 
culture  by  their  words  and  actions.  Personal  involvement  in  the  performance  initiative 
is  required  to  demonstrate  commitment,  understanding,  and  the  company’s  values.”  Is 
leadership  really  involved? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


“  Continual  improvement  and  learning :  Throughout  the  Criteria,  the  last  area  to  address 
is  often  “How  do  you  evaluate  and  improve  (the  process  just  described)?” 

“Ongoing  improvement  of  key  processes  is  essential  to  achieve  or  maintain  a  leading 
position  in  the  market.  ”  Is  this  a  real  ongoing  effort  to  improve  processes? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 
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TABLE  le 

CULTURAL  CHANGE  SURVEY  PERCEPTIONS2 


4.  “ Employee  participation  and  development:  Customer  delight  is  the  goal  defined  by  the 

Baldrige  Criteria.  One  of  the  ways  to  achieve  this  is  to  develop  the  fall  potential  of  the 
workforce.  This  includes  employee  involvement,  training,  recognition,  safety,  and 
satisfaction.”  Is  employee  development ,  along  these  lines,  progressing? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


5.  “ Fast  response :  Success  in  the  competitive  marketplace  requires  efficient  and  effective 

processes.  Reducing  the  time  it  takes  to  bring  a  new  product  or  service  to  the 
marketplace  is  now  a  recognized  objective.  Likewise,  time  to  respond  to  a  customer 
inquiry  or  complaint  or  time  to  respond  to  an  employee  suggestion  often  differentiates 
the  high-performance  organizations  from  the  bureaucratic  ones.”  How  well  do  we 
respond  to  customer  requirements  in  terms  of  response  time? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


6.  “ Design  quality:  Products  and  services  that  are  designed  to  be  insensitive  to  expected 

variation  (robust)  give  a  company  an  advantage.  Success  will  be  achieved  even  when 
conditions  are  not  optimal.  Moving  the  focus  of  quality  from  manufacturing  or  delivery 
to  the  design  stage  allows  the  biggest  quality  impact  on  the  overall  product  or  service.  ” 
What  is  your  impression  of  the  design  quality  in  the  district? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 
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7.  “ Prevention :  Intuitively,  it  is  accepted  that  prevention  is  more  effective  than  fire 

fighting.  But  it  is  hard  to  talk  to  someone  about  fire  prevention  when  their  house  is  on 
fire.  If  we  don’t  make  time  for  prevention  activities,  the  fires  will  continue.”  Are  we 
too  fire-fight  oriented  or  can  improvements  be  made? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


8.  “ Partnership  development-.  Partnerships  between  labor  and  management,  between 

suppliers  and  the  company,  and  between  the  company  and  customers  offer  win-win 
opportunities.  Partnerships  with  educational  institutions  to  develop  the  workforce  and 
partnership  with  other  companies  for  joint  developments  are  examples  of  other  win-win 
opportunities.”  Is  this  a  management  attitude  about  partnership  development? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


9.  “ Long-range  outlook :  Developing  partnerships  with  suppliers,  developing  the  full 

potential  of  the  workforce,  and  creating  the  future  for  our  customers  require  a  long- 
range  outlook.  At  least  3-5  years  and,  in  some  industries  10-15  years,  should  be  the 
planning  horizon.”  Are  we  actually  developing  long-range  plans  and  partnership 
programs  with  external  suppliers? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 
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TABLE  le 

CULTURAL  CHANGE  SURVEY  PERCEPTIONS2 


10.  “Management  by  fact:  Facts  combined  with  product  or  process  knowledge  are  very 
powerful.  Trends,  projects,  and  cause-and-effect  relationships  should  be  studied  with 
an  understanding  of  variation.”  Is  there  a  positive  ongoing  effort  to  collect  and  use 
data  and  information  for  decision  making? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


11.  “ Good  corporate  citizen :  A  company’s  quality  goals  and  objectives  should  address 
corporate  responsibility  and  citizenship  such  as  public  health,  public  safety,  and  the 
environment.  Giving  back  to  the  community  by  working  with  educational  institutions 
or  local  government  are  included  in  the  Criteria.”  Does  your  organization  present  a 
positive  image  in  this  area? 


Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


12.  “Results  orientation :  Internal  results  should  be  correlated  with  field  results  and  financial 
indicators.  Performance  indicators  are  a  means  of  communicating  and  monitoring 
progress.  These  include  customer  satisfaction  and  retention,  market  share,  product  and 
service  quality,  human  resource  performance  and  development,  financial  indicators,  and 
public  responsibility.”  Does  your  organization  have  a  “results  orientation”? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 
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TABLE  le 

CULTURAL  CHANGE  SURVEY  PERCEPTIONS2 


13.  The  Baldrige  Award  is  a  very  prestigious  achievement  placing  an  organization  among 
“the  best” .  What  level  of  effort  do  you  believe  your  organization  is  generating  to 
actually  attempt  to  win  this  award? 

0-20%  20-40%  40-60%  60-80%  80-100% 

1  2  3  4  5 


14.  Self-empowered  teams  have  a  reputation  for  being  the  best  environments  to  work  in  and 
the  most  productive.  Where  does  your  organization  stand  in  this  regard? 

Very  Very 

Poor  Poor  Moderate  Good  Excellent 

1  2  3  4  5 


TABLE  2e 

DEVELOPMENT  OF  FREQUENCY  DISTRIBUTION  OF  PERCEPTIONS 


(2) 

Response 

Not  at  all 

Very  little 

To  some  degree 

Very  positive 
strides 

Being  given  full 
support 


Total  Sample 


(3) 

Frequencies 


(5) 

Calc.  Or  Mea 


Mean  Score  142/49 

=  2.89 

Mode  Score 

=  3.00 

Median 

=  3.00 

Response  Code 
Figure  2e. 
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The  reader  can  observe  that  indeed 
response  three  is  the  most  popular  but  that 
there  is  a  spread  of  responses  around  this 
central  value.  Previous  discussions 
concerning  frequency  distributions  can  be 
used  to  further  statistically  describe  the 
significance  of  this  distribution.  In  this  case 
it  is  not  necessary  to  grind  through  the 
statistical  calculations  to  determine  the 
variance  or  standard  deviations,  etc.  This 
type  of  distribution  analysis  does  not  warrant 
a  great  deal  of  statistical  analysis.  The  shape 
of  the  distribution,  mean,  mode  and  median 
is  sufficient  to  tell  the  analysts  what  is 
happening  and  what  the  perceptions  of  the 
customers  are. 

Figure  le  will  aid  in  deciphering  the 
significance  of  the  various  shapes  this 
distribution  could  have  taken.  It  will  aid  the 
user  in  the  event  that  different  shapes  do 
arise  with  other  questions  and  other  surveys 
taken  in  the  future. 

The  shape  of  curve  #2  tells  us  that  the 
responses  are  fairly  normal,  with  a  centered 
peak  and  symmetrical  in  shape.  This  is  a 
normal  shape  in  that  responses  show  a  spread 
but  the  normal  “bell  shaped”  geometry  is 
present.  Curve  #1  is  normal  in  symmetry  but 
note  that  it  is  far  more  peaked  than  curve  #2. 
This  means  there  is  less  spread  among  the 
respondents— they  tend  to  agree  more 
strongly  than  those  indicated  by  curve  ft 2 . 
Curve  three  is  also  normal  in  shape  and  holds 
the  “bell”  shape  but  is  indicative  of  a  wider 
disagreement  among  the  respondents  as  noted 
by  the  lower,  flatter  appearance.  Curve  #4 
lost  its  ell  configuration  and  is  skewed  to  the 
right.  This  implies  that  the  respondents  are 
leaning  toward  an  answer  to  the  right  of  the 
curve  and  have  decidedly  shied  away  from 
whatever  the  response  to  the  right  of  the 


curve  indicated.  Curve  #5  is  similar  to  curve 
#4  but  is  asymmetrical  to  the  right.  This 
means  that  whatever  responses  were  to  the 
right  of  the  question  are  more  highly  favored 
than  those  to  the  left.  Curve  #6  says  that  the 
respondents  are  polarized— about  half 
favoring  a  response  to  the  left  of  the  question 
and  half  to  the  right.  This  entails  a  split  of 
an  organization  in  outlook  or  views 
concerning  the  question.  Curve  #1  is  an 
extreme  of  curve  #6.  Here,  there  is  extreme 
disagreement  and,  one  would  go  so  far  as  to 
say,  a  major  split,  rift  or  high  degree  of 
difference  in  views  on  this  issue.  This 
degree  of  separation  between  peaks  probably 
borders  on  hostility— a  hot  issue  is  at  stake. 
Curve  #8  is  a  little  like  curve  #6  but  there  is 
a  polarization  between  a  minority  view  and  a 
majority  view  on  the  question  at  hand,  as 
depicted  by  the  different  peak  sizes.  Finally, 
curve  #9  shows  an  even  spread  of  views  and 
responses  throughout  the  organization. 
Employees  or  customers  are  perhaps 
“indifferent”  or  the  issue  is  irrelevant. 

As  the  reader  can  see,  the  shape  of  the 
curve  developed  or  formed  from  plotting  a 
frequency  curve  can  tell  a  good  deal  about 
the  population  surveyed.  It  becomes  clear 
that  the  questions  in  the  survey  must  be 
unbiased  and  very  clear  because 
interpretation  can  be  distorted  by  a 
misunderstood  question.  It  is  also  wise  to 
pre-test  a  questionnaire  or  survey  using  a 
small  population  in  order  to  weed  out 
potential  problems  with  question  language. 

There  are  variations  in  the  construction  of 
frequency  distribution  curves.  Notice  that 
question  13  in  Table  le  uses  a  range  of 
values  for  answers.  Some  questions  will 
require  this  approach.  Notice  the  above 
example  illustrates  the  application  of  a 
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Figure  le.  -  Interpretation  of  Frequency  Distribution  Shapes 
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process  which  has  widespread  acceptance  and 
a  good  deal  of  utility.  There  are  questions 
which  require  yes/no,  true/false,  fill  in  the 
blank,  short  answers,  multiple  choice 
answers,  etc.,  and  all  of  them  are  valuable. 
This  approach  provides  a  good  deal  of 
information.  There  is  an  alternative  way  to 
design  a  questionnaire.  Ask  the  customer  to 
construct  a  list  of  survey  questions  that 
he/she  would  want  you  to  ask  them  about 
your  performance  at  the  end  of  some 
performance  period;  say  a  year.  The 
advantages  of  this  are  obvious. 

Pareto  Charts 

A  Pareto  chart  shows  the  relative 
frequency  of  importance  of  collected 
information  or  data.  Although  it  is  a 
relatively  simple  tool  and  among  the  seven 
key  tools  of  TQM  [Rodakowski,  1994],  it  is 
valuable  enough  to  include  in  a  discussion  of 
performance  measurement.  The  approach 
and  deployment  of  this  tool  is 
straightforward.  Whenever  a  problem  arises 
and  solutions  are  sought,  the  forum  is  asked 


to  prioritize  solutions  to  the  problem.  The 
actual  ranking  of  priorities  can  be  achieved 
through  focus  groups,  brainstorming, 
consensus  resolution,  win-win  bargaining, 
majority  vote,  Delphi  method,  or  some  other 
satisfactory  method.  Figure  3e  illustrates 
the  process.  There  are  variations  of  the 
ranking  process  itself;  ranking  by  vote, 
ranking  by  use  of  actual  data,  the  ranking  of 
solutions,  the  ranking  of  the  “core”  or  root 
problems,  et  al.  The  ranking  is  in  order  of 
importance  to  the  situation  being  addressed, 
be  it  a  problem  or  a  solution.  It  requires 
agreement  and  discussion  of  difficult  issues, 
it  demands  systems  thinking,  requires 
planning  and  coordination,  represents  doing 
more  with  less,  it  forces  prioritization,  and, 
most  of  all,  it  compels  action.  It  also  ranks 
performance,  if  designed  to  do  so.  As  Figure 
3e  shows,  a  group  (PAT  team,  ESC,  etc.) 
has  determined  that  50%  of  wasted  time  in 
cycle  time  analysis  came  from  waiting  time, 
another  30%  came  from  rework,  15%  came 
from  review  time,  and  the  remainder  (5%) 
came  from  a  variety  of  sources.  It  is  clear 
what  must  be  done  and  where  to  start. 


Appendix  E  -  Surveys  &  Pareto  Charts  - 181 


PARETO  CURVE 


P 

E 

R 

C 

E 

N 

T 


Accumulative 


ATTRIBUTES 


Figure  3e.  -  Pareto  Curve 
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E.2.  SUMMARY 


This  book  has  concerned  itself  with 
the  measurement  and 
improvement  of  performance.  It 
is  applicable  to  local,  state  and  federal 
governments  as  well  as  other  non-profit  and 
commercial  organizations.  It  contains  state- 
of-the-art  quantitative  methodologies  in 
support  of  continuous  improvement 
initiatives,  TQM,  and  the  APIC  tradition.  It 
not  only  explains  “how”  to  measure 
organizational  performance  but,  in  a  macro 
context,  “what”  to  measure  as  well. 

It  began  with  a  brief  discussion  of  the 
new  laws  and  executive  orders  now 
permeating  the  Federal  government  such  as 
NPR,  GPRA,  CSS,  APIC,  and  Labor- 
Management  Partnership  Agreements~all 
part  of  a  growing  national  performance 
movement.  It  then  detailed  such  timely 
topics  as  waste  evaluation,  cycle-time 
analysis,  value-added,  business  planning, 
benchmarking,  productivity  measurement, 
efficiency  measurement,  statistical  process 
control,  process  and  systems  mapping, 
Shewhart  and  Demos  control  charts,  quality 
function  deployment  and  a  host  of  additional 
up-to-date  techniques  and  methods.  The  text 
also  included  outlines  of  such  organizational 
attributes  as  structure,  the  need  and 
development  of  a  vision  and  mission 
statement,  goals  and  objectives,  operating 
characteristics,  key,  core  or  critical  processes 
among  others. 


The  text’s  strength  lies  in  applied 
examples  covering  such  areas  as  resource 
inputs,  processes  or  resource  transformation, 
product  and  service  outputs,  outcomes, 
program  effectiveness,  etc.  The  performance 
measures  are  explained  using  detailed 
examples  mimicking  real  world  situations. 
The  material  is  comprehensive  and  technical 
but  necessary  if  metrics  are  to  become  part  of 
the  cultural  process  change  needed  to 
overcome  effectiveness,  efficiency  and 
productivity  problems.  Unusual  but 
important  areas  are  also  covered.  The  world 
is  rapidly  becoming  more  competitive  with 
increased  needs  and  importance  in  areas  of 
change  adaptation,  knowledge  acquisition  and 
capability,  and  information  processing  and 
infusion.  Performance  measurement  for 
these  new  business  characteristics  is  also 
embraced. 

There  are  many  tools  not  covered  which 
are  of  a  more  traditional  vein  but  still  very 
valid.  They  can  be  found  in  the  material 
listed  in  the  bibliography.  Important  tools 
and  methods  such  as  simple  and  multi  variate 
regression  analysis,  Monte  Carlo 
simulations,  correlations  analysis,  “t”  test  of 
significance,  analysis  of  variance,  Chi 
Square,  etc.  are  also  part  of  the  new  analyst’s 
tool  chest.  These  are  no  longer  tools 
reserved  for  the  statistician;  but  their 
application  is  now  a  requirement  among 
“world  class”  organizations. 
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Activities.  The  steps  of  a  process.  [James  H.  Saylor]. 

Activity -based  cost  accounting  (or  management).  A  form  of  managerial  accounting  in  which 
indirect  costs,  normally  thought  of  as  overhead,  are  assigned  to  projects  or  activities  rather 
than  to  aggregations  of  type  of  costs  (travel,  supplies,  consultants)  and  are  apportioned  by  an 
appropriate  “driver”  rather  than  just  becoming  “labor  burden”  as  in  standard  accounting. 
[Thor,  1994] 

Adaptability.  The  flexibility  of  the  process  to  handle  future,  changing  customer  expectations  and 
today’s  individual,  special  customer  requests.  It  is  managing  the  process  to  meet  today’s 
special  needs  and  future  requirements.  Adaptability  is  an  area  largely  ignored,  but  it  is  critical 
for  gaining  a  competitive  edge  in  the  marketplace.  Customers  always  remember  how  you 
handled,  or  didn’t  handle,  their  special  needs.  [Harrington,  1991] 

Benchmarking.  Systematic  comparison  of  elements  for  the  performance  of  an  organization 
against  that  of  other  organizations,  usually  with  the  aim  of  mutual  improvement.  [Thor,  1994] 

Best  of  class.  One  of  a  group  of  similar  organizations  whose  overall  performance,  effectiveness, 
efficiency,  and  adaptability  is  superior  to  all  others.  [James  H.  Saylor,  1992] 

Capital  productivity.  The  output  of  a  process  divided  by  the  quantity  of  capital  associated  with  the 
creation  or  modification  of  that  output,  thus  a  measure  of  the  efficiency  or  effectiveness  of  the 
use  of  capital.  [Thor,  1994] 

Common  Cause.  The  source  of  random,  or  natural,  variation  in  a  process.  Common  causes— or 
“assignable  causes,”  as  they  are  sometimes  called— are  inherent  in  the  production  process  itself. 
[Hart,  Bogan,  1992) 

Continuous  improvement  system.  A  disciplined  methodology  to  achieve  the  goal  of  commitment 
to  excellence  by  continually  improving  all  processes.  [James  H.  Saylor,  1992] 

Control  chart.  In  quality  analysis,  a  charting  technique  that  allows  determination  of  whether  the 
variation  being  observed  in  a  process  is  within  “normal”  bounds  or  a  symptom  of  a  changed 
process.  [Thor,  1994] 

Cycle  time.  The  elapsed  time  it  takes  to  do  a  defined  piece  of  work,  from  the  receipt  of  all  needed 
supplies  and  information  to  the  delivery  of  the  finished  work.  [Thor,  1994] 
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Denting,  W.  Edwards.  (1900-1994)  A  pioneer  of  the  modem  quality  movement  who  emphasized 
the  use  of  statistical  techniques  and  disciplined  striving  for  zero  defects.  [Thor,  1994] 

Downtime.  The  cost  to  redo  rather  than  to  go  on  to  the  next,  new  task.  [Roseland,  1989] 

Effect.  A  problem  or  defect  that  occurs  on  the  specific  job  to  which  a  group  or  team  is  assigned. 
[James  H.  Saylor,  1992] 

Effectiveness.  (1)  The  extent  to  which  the  outputs  of  the  process  or  subprocess  meet  the  needs  and 
expectations  of  its  customers.  A  synonym  for  effectiveness  is  quality.  Effectiveness  is  having 
the  right  output  at  the  right  place,  at  the  right  time,  at  the  right  price.  Effectiveness  impacts  the 
customer.  [Harrington,  1991]  (2)  A  characteristic  used  to  describe  a  process  in  which  output 
conforms  to  requirements.  [James  H.  Saylor,  1992] 

Efficiency.  (1)  The  extent  to  which  resources  are  minimized  and  waste  is  eliminated  in  the  pursuit 
of  effectiveness.  Productivity  is  a  measure  of  efficiency.  [Harrington,  1991].  (2)  A  characteristic 
used  to  describe  a  process  that  produces  the  required  output  at  a  perceived  minimum  cost.  [James 
H.  Saylor,  1992] 

Empowerment.  The  power  of  people  to  do  whatever  is  necessary  to  do  the  job  and  improve  the 
system.  [James  H.  Saylor,  1 992] 

Energy  productivity.  The  relationship  between  the  output  of  a  process  and  the  amount  of  energy 
required  to  create  or  modify  that  output.  [Thor,  1994] 

Facilitator.  One  who  assists  the  group  or  team  in  applying  the  TQM  tools  and  techniques.  [James 
H.  Saylor,  1992] 

Family  of  measures.  A  balanced  collection  of  four  to  six  performance  measures  (usually  including 
productivity,  quality,  and  customer  satisfaction)  that  together  provide  a  comprehensive  view  of 
organizational  results  but  individually  also  provide  diagnostic  value.  [Thor,  1994] 

Feedback.  A  process  in  which  the  producers  of  a  result  receive  modifying  information  from  that 
result.  In  this  case,  individuals  and  work  groups  of  all  types  receive  direct  information  about 
their  outputs.  [Thor,  1994] 

Flowchart.  A  step-by-step  display  of  how  a  process  functions  using  a  series  of  coded  figures  to 
represent  different  kinds  of  activities  and  interventions,  used  in  business  process  reengineering 
and  benchmarking  data  development.  [Thor,  1 994] 

Functional  organization.  An  organization  responsible  for  a  major  organizational  function,  such  as 
marketing,  sales,  design,  manufacturing,  or  distribution.  [James  H.  Saylor,  1992] 
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Goal.  The  specific  desired  outcome.  [James  H.  Saylor,  1992] 

Hierarchical  nature  of  a  process.  The  various  levels  of  a  process.  [James  H.  Saylor,  1992] 

Histogram.  A  set  of  vertical  bar  graphs  that  demonstrates  the  distribution  of  a  variable  in  a 
population.  [Thor,  1994] 

Index.  A  set  of  numbers  showing  percentage  variation  from  an  arbitrary  standard,  usually  100, 
representing  the  status  at  some  earlier  time.  [Thor,  1 994] 

Inflation(deflation) .  The  degree  to  which  general  or  specific  prices  have  increased  (or  decreased) 
from  one  time  period  to  another.  [Thor,  1 994] 

Input.  That  which  is  used  to  make  an  output;  in  productivity  measurement  an  expression  of  the 
physical  amount  (or  dollar  value)  of  one  or  more  elements  utilized  in  the  production  process. 
[Thor,  1994] 

Kaizen.  A  Japanese  expression  referring  to  continuous  improvement  in  all  phases  of  a  business. 
[Hart,  Bogan,  1992] 

Key  process.  A  series  of  activities  so  critical  that  value  added  work  cannot  be  accomplished  without 
them. 

Key  result  area.  An  area  of  resource  input,  processing  activities,  output,  outcome,  or  results  which, 
when  measured,  serve  to  diagnose  an  entire  process  or  system. 

Labor  productivity.  The  relationship  between  the  output  of  a  process  or  entity  and  the  labor  inputs 
used  in  creation  or  modification  of  that  output.  [Thor,  1994] 

Malcolm  Baldrige  National  Quality  Award.  A  quality  award  first  presented  by  the  President  of  the 
United  States  in  1988  that  recognizes  approach,  deployment,  and  results  of  a  manufacturing  or 
service  organization’s  quality  initiative.  Its  basic  criteria  have  become  the  standard  for  self- 
assessment  of  quality  in  the  United  States.  [Thor,  1994] 

Materials  productivity.  The  relationship  between  the  output  of  a  process  and  the  materials  inputs 
used  in  creation  or  modification  of  that  output.  [Thor,  1994] 

Measurement.  The  act  or  process  of  measuring  in  order  to  compare  results  with  requirements.  A 
quantitative  estimate  of  performance.  (James  H.  Saylor,  1992] 

Multifunctional  team.  A  team  consisting  of  representatives  from  more  than  one  function.  [James 
H.  Saylor,  1992] 
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Outcome.  Downstream  from  output,  an  outcome  is  the  ultimate  result  of  a  productive  process.  If 
the  output  is  a  high  school  diploma,  the  outcome  is  a  job  offer.  If  the  output  is  a  patient 
diagnosed,  the  outcome  is  a  patient  cured.  [Thor,  1994] 

Pareto  analysis.  A  system  of  analysis  based  on  the  principle  that,  in  any  phenomenon,  relatively  few 
factors  account  for  the  majority  of  effects.  Juran  uses  the  phrase  “vital  few”  to  suggest  that  it  is 
more  efficient  and  less  costly  to  concentrate  on  the  most  important  sources  or  types  of  failures, 
customers,  and  so  on.  [Hart,  Bogan,  1992] 

Pareto  chart.  A  chart  that  connects  frequency  of  process  difficulties  with  potential  cause 
classifications.  Pareto’s  law  suggests  that  a  large  percentage  of  the  difficulties  are  typically 
caused  by  a  small  percentage  of  the  causes.  [Thor,  1994] 

Performance  measurement.  A  means  of  evaluating  efficiency,  effectiveness,  and  results.  A 
balanced  performance  measurement  scorecard  includes  financial  and  nonfinancial  measures 
focusing  on  quality,  cycle  time,  and  cost.  Performance  measurement  should  include  program 
accomplishments  in  terms  of  outputs  (quantity  of  products  or  services  provided,  e.g.,  how  many 
items  efficiently  produced?)  And  outcomes  (results  of  producing  outputs,  e.g.,  are  outputs 
effectively  meeting  intended  agency  mission  objectives?).  [Management  Concepts  Inc.,  1995] 

Process.  (1)  A  systematic  series  of  actions  directed  to  the  achievement  of  a  goal.  [J..M.  Juran,  1992] 
(2)  A  series  of  activities  that  takes  an  input,  modifies  the  input  (work  takes  place/value  is  added), 
and  produces  an  output.  [James  H.  Saylor,  1992] 

Process  analysis.  Tool  used  to  improve  a  process  and  reduce  process  time  by  eliminating  non-value- 
added  activities  and/or  simplifying  the  process.  [James  H.  Saylor,  1992] 

Process  capability.  The  ability  of  a  process  to  meet  operating  goals  or  internal— or  external-customer 
requirements.  “Capability”  may  differ  from  actual  performance  due  to  “special  causes”— 
conditions  or  events  due  purely  to  chance  and  not  to  the  production  system  itself.  [Hart,  Bogan, 
1992] 

Process  control.  The  systematic  evaluation  of  performance  of  a  process,  and  the  taking  of  corrective 
action  in  the  event  of  nonconformance.  [J.M.  Juran,  1992] 

Process  improvement.  The  set  of  activities  used  to  detect  and  remove  common  causes  of  variation 
in  order  to  improve  process  capability.  Process  improvement  leads  to  quality  improvement. 
[James  H.  Saylor,  1992] 

Process  management.  A  management  approach  comprising  quality  management  and  process 
optimization.  [James  H.  Saylor,  1992] 
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Process  performance.  A  measure  of  the  effectiveness  and  efficiency  with  which  a  process  satisfies 
customer  requirements.  [James  H.  Saylor,  1992] 

Productivity.  The  relationship  between  the  output  of  a  process  or  entity  and  one  or  more  of  the 
inputs  used  to  create  that  output.  [Thor,  1994] 

QC,  Quality  Circle.  A  group  of  workers  who  have  undergone  training  for  the  purpose  of  solving 
work-related  problems.  [J.M.  Juran,  1992] 

Quality.  (1)  “Quality  is  meeting  customer  needs  and  expectations  consistently  and  efficiently.” 
[Turner,  1993]  (2)  Product  features  that  respond  to  customer  needs;  (3)  freedom  from 
deficiencies.  [J.M.  Juran,  1 992] 

Re-analysis.  The  cost  of  correcting  defects.  [Roseland,  1989] 

Replacement  cost.  What  it  would  cost  to  replace  a  specified  amount  of  old  productive  capacity 
using  current  design  and  technology.  [Thor,  1994] 

Review.  The  cost  of  another  review  cycle.  [Roseland,  1989] 

Robust  design.  The  design  of  a  product  for  minimal  quality  losses.  [James  H.  Saylor,  1992] 

Root  cause.  Underlying  reason  for  nonconformance  within  a  process.  When  it  is  removed  or 
corrected,  the  nonconformance  is  eliminated.  [James  H.  Saylor,  1992] 

Simulation.  Techniques  of  observing  and  manipulating  an  artificial  mechanism  (model)  that 
represents  a  real-world  process  that,  for  technical  or  economical  reasons,  is  not  suitable  or 
available  for  direct  experimentation.  [James  H.  Saylor,  1992] 

Special  cause.  An  “abnormal”  source  of  variation  that  does  not  arise  from  the  production  process 
itself,  but  which  is  extraneous  and  unpredictable.  [Hart,  Bogan,  1992] 

Standard  deviation.  A  parameter  describing  the  spread  of  the  process  output.  The  positive  square 
root  of  the  variance.  [James  H.  Saylor,  1992] 

Statistical  control.  Term  used  to  describe  a  process  from  which  all  special  causes  of  variation  have 
been  removed,  leaving  only  common  causes.  Such  a  process  is  also  said  to  be  stable.  [James 
H.  Saylor,  1992] 

Statistical  Process  Control  (SPC).  Based  on  the  principle  that  no  two  units  of  output  of  a  process 
are  likely  to  have  the  exact  same  specifications.  SPC  involves  the  mathematical  determination 
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of  acceptable  limits  of  variation.  Graphs  are  used  by  workers  to  plot  output  variable  and  visually 
determine  when  a  process  is  “in”  or  “out  of’  control.  [Hart,  Bogan,  1992] 

Strategy.  A  broad  course  of  action,  chosen  from  a  number  of  alternatives,  to  accomplish  a  stated 
goal  in  the  face  of  uncertainty.  [James  H.  Saylor,  1 992] 

Stretch  goal.  An  ambitious,  usually  long-term  quality  goal  that  requires  extraordinary  effort, 
innovation,  and  planning  to  achieve.  [Hart,  Bogan,  1992] 

Subprocesses.  The  internal  processes  that  make  up  a  process.  [James  H.  Saylor,  1992] 

System.  A  configuration  of  things  or  processes  so  inter-connected  as  to  behave  as  one. 
[Rodakowski,  1994] 

Total  factor  productivity.  Sometimes  used  interchangeably  with  total  productivity,  but  more 
precisely  the  relationship  of  a  value-added  output  to  its  labor  and  capital  inputs  with  price  effects 
removed.  [Thor,  1994] 

Total  productivity.  The  relationship  between  the  output  of  a  process  and  all  of  the  inputs  to  that 
process  (typically  labor,  capital,  energy,  and  materials)  with  price  effects  removed.  [Thor,  1994] 

Total  Quality  Management  (TQM).  A  philosophy  and  a  set  of  guiding  principles  that  represent  the 
foundation  of  a  continuously  improving  organization.  TQM  is  the  application  of  quantitative 
methods  and  human  resources  to  improve  the  material  services  supplied  to  an  organization,  all 
the  processes  within  the  organization,  and  the  degree  to  which  the  needs  of  the  customer  are  met, 
now  and  in  the  future.  TQM  integrates  fundamental  management  techniques,  existing 
improvement  efforts,  and  technical  tools  under  a  disciplined  approach  focused  on  continuous 
improvement.  [James  H.  Saylor,  1992] 

Value  added.  Sales  less  purchased  goods  and  services,  or  viewed  from  the  other  direction,  labor 
plus  capital  plus  profit.  [Thor,  1994] 

Variance.  In  quality  management  phraseology,  any  deviation  from  specifications;  in  statistics,  the 
square  of  the  standard  deviation. 

Waste.  The  net  loss  in  labor  and  material.  [Roseland,  198] 

Weighting.  The  systematic  combining  of  unlikes,  in  this  case  (1)  the  combining  of  process  outputs 
that  have  different  inherent  degrees  of  input  usage  and  (2)  the  combining  of  the  results  of  the 
family  of  measures  members  based  on  predetermined  relative  importance.  [Thor,  1994] 
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